Skip to content

Java 8 Stream Documentation improvements #298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Feb 11, 2019
Merged

Java 8 Stream Documentation improvements #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e56bf90
Cleaned up whitespace at end of line.
gmessner Feb 10, 2019
1c0c62b
Added test for lazy Stream evaluation (#295).
gmessner Feb 10, 2019
d7027eb
Cleaned up Stream support documentaion (#295).
gmessner Feb 10, 2019
c332ee0
Fixed typos and added info on lazy eval Streams (#295).
gmessner Feb 11, 2019
470caa3
Changed testStreamLimit() to use a lazy stream (#295).
gmessner Feb 11, 2019
6703902
Fixed mis-placed brace (#295).
gmessner Feb 11, 2019
3e28608
Clarified info on streams (#295).
gmessner Feb 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 30 additions & 23 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ To utilize the GitLab API for Java in your project, simply add the following dep
```java
dependencies {
...
compile group: 'org.gitlab4j', name: 'gitlab4j-api', version: '4.9.15'
compile group: 'org.gitlab4j', name: 'gitlab4j-api', version: '4.9.16'
}
```

Expand All @@ -22,7 +22,7 @@ dependencies {
<dependency>
<groupId>org.gitlab4j</groupId>
<artifactId>gitlab4j-api</artifactId>
<version>4.9.15</version>
<version>4.9.16</version>
</dependency>
```

Expand Down Expand Up @@ -141,34 +141,41 @@ List<Project> allProjects = projectPager.all();

---
## Java 8 Stream Support
As of GitLab4J-API 4.9.2, you can also stream list based items in a Java 8 Stream using a Pager instance as follows:
```java
// Get a Pager instance to get a Stream<Project> instance.
Pager<Project> projectPager = gitlabApi.getProjectsApi().getProjects(10);
As of GitLab4J-API 4.9.2, all GitLabJ-API methods that return a List result also similarlly named method returns a Java 8 Stream. The Stream returning methods use the following naming convention: ```getXxxxxStream()```.


**IMPORTANT**
The built-in methods that return a Stream do so using ___eager evaluation___, meaning all items are pre-fetched from the GitLab server and a Stream is returned which will stream those items. **Eager evaluation does NOT support paralell reading of data from ther server, it does however allow for paralell processing of the Stream post data fetch.**

To stream using ___lazy evaluation___, use the GitLab4J-API methods that return a ```Pager``` instance, and then call the ```lazyStream()``` method on the ```Pager``` instance to create a lazy evaluation Stream. The Stream utilizes the ```Pager``` instance to page through the available items. **A lazy Stream does NOT support parallel operations or skipping.**


**Eager evaluation example usage:**

// Stream the Projects printing out the project name.
projectPager.stream().map(Project::getName).forEach(name -> System.out.println(name));
```
The following API classes also include ```getXxxxxStream()``` methods which return a Java 8 Stream:
```
GroupApi
IssuesApi
NotesApi
ProjectApi
RepositoryApi
TagsApi
UserApi
```
Example usage:
```java
// Stream the visible Projects printing out the project name.
gitlabApi.getProjectsApi().getProjectsStream().map(Project::getName).forEach(name -> System.out.println(name));
// Stream the visible projects printing out the project name.
Stream<Project> projectStream = gitlabApi.getProjectApi().getProjectsStream();
projectStream.map(Project::getName).forEach(name -> System.out.println(name));

// Operate on the stream in parallel, this example sorts User instances by username
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is a good idea to give such an example here. Parallel operations are generally used for performance optimization, but here project gathering is so slow, that this operation doesn't add any improvement. As a result, the user can be confused that this operation can be super fast and this library can fetch data in multiple channels.

Copy link
Collaborator Author

@gmessner gmessner Feb 11, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parallel operations are often performed after fetching the data, parallel operations are not only used to speed up fetching, I have added more info to make the fact that reading of the data for the Stream cannot be done in parallel.

// NOTE: Fetching of the users is not done in paralell,
// only the soprting of the users is a paralell operation.
Stream<User> stream = new UserApi(gitLabApi).getUsersStream();
List<User> sortedUsers = stream.parallel().sorted(comparing(User::getUsername)).collect(toList());
List<User> users = stream.parallel().sorted(comparing(User::getUsername)).collect(toList());
```

**Lazy evaluation example usage:**

```java
// Get a Pager instance to that will be used to lazily stream Project instances.
// In this example, 10 Projects per page will be pre-fetched.
Pager<Project> projectPager = gitlabApi.getProjectApi().getProjects(10);

// Lazily stream the Projects, printing out each project name, limit the output to 5 project names
projectPager.lazyStream().limit(5).map(Project::getName).forEach(name -> System.out.println(name));
```


---
## Java 8 Optional&lt;T&gt; Support
GitLab4J-API supports Java 8 Optional&lt;T&gt; for API calls that result in the return of a single item. Here is an example on how to use the Java 8 Optional&lt;T&gt; API calls:
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/gitlab4j/api/GitLabApi.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ public class GitLabApi {
private final static Logger LOGGER = Logger.getLogger(GitLabApi.class.getName());

/** GitLab4J default per page. GitLab will ignore anything over 100. */
public static final int DEFAULT_PER_PAGE = 100;
public static final int DEFAULT_PER_PAGE = 96;

/** Specifies the version of the GitLab API to communicate with. */
public enum ApiVersion {
Expand Down Expand Up @@ -1393,7 +1393,7 @@ public static final <T> T orElseThrow(Optional<T> optional) throws GitLabApiExce

return (optional.get());
}

/**
* Gets the SnippetsApi instance owned by this GitLabApi instance. The SnippetsApi is used
* to perform all snippet related API calls.
Expand Down
89 changes: 59 additions & 30 deletions src/main/java/org/gitlab4j/api/Pager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@
import com.fasterxml.jackson.databind.ObjectMapper;

/**
* <p>This class defines an Iterator implementation that is used as a paging iterator for all API methods that
* <p>This class defines an Iterator implementation that is used as a paging iterator for all API methods that
* return a List of objects. It hides the details of interacting with the GitLab API when paging is involved
* simplifying accessing large lists of objects.</p>
*
*
* <p>Example usage:</p>
*
*
* <pre>
* // Get a Pager instance that will page through the projects with 10 projects per page
* Pager&lt;Project&gt; projectPager = gitlabApi.getProjectsApi().getProjectsPager(10);
Expand All @@ -35,8 +35,8 @@
* System.out.println(project.getName() + " : " + project.getDescription());
* }
* }
* </pre>
*
* </pre>
*
* @param <T> the GitLab4J type contained in the List.
*/
public class Pager<T> implements Iterator<List<T>>, Constants {
Expand All @@ -48,6 +48,7 @@ public class Pager<T> implements Iterator<List<T>>, Constants {

private List<String> pageParam = new ArrayList<>(1);
private List<T> currentItems;
private Stream<T> pagerStream = null;

private AbstractApi api;
private MultivaluedMap<String, String> queryParams;
Expand All @@ -59,7 +60,7 @@ public class Pager<T> implements Iterator<List<T>>, Constants {

/**
* Creates a Pager instance to access the API through the specified path and query parameters.
*
*
* @param api the AbstractApi implementation to communicate through
* @param type the GitLab4J type that will be contained in the List
* @param itemsPerPage items per page
Expand Down Expand Up @@ -114,7 +115,7 @@ public class Pager<T> implements Iterator<List<T>>, Constants {

/**
* Get the specified integer header value from the Response instance.
*
*
* @param response the Response instance to get the value from
* @param key the HTTP header key to get the value for
* @return the specified integer header value from the Response instance
Expand All @@ -136,7 +137,7 @@ private int getHeaderValue(Response response, String key) throws GitLabApiExcept

/**
* Sets the "page" query parameter.
*
*
* @param page the value for the "page" query parameter
*/
private void setPageParam(int page) {
Expand Down Expand Up @@ -204,7 +205,7 @@ public List<T> next() {

/**
* This method is not implemented and will throw an UnsupportedOperationException if called.
*
*
* @throws UnsupportedOperationException when invoked
*/
@Override
Expand Down Expand Up @@ -289,7 +290,7 @@ public List<T> page(int pageNumber) {
throw new RuntimeException(e);
}
}

/**
* Gets all the items from each page as a single List instance.
*
Expand All @@ -312,34 +313,62 @@ public List<T> all() throws GitLabApiException {
}

/**
* Builds and returns a Stream instance for streaming all the items from each page.
* Builds and returns a Stream instance which is pre-populated with all items from all pages.
*
* @return a Stream instance for streaming all the items from each pag
* @throws GitLabApiException if any error occurs
* @return a Stream instance which is pre-populated with all items from all pages
* @throws IllegalStateException if Stream has already been issued
* @throws GitLabApiException if any other error occurs
*/
public Stream<T> stream() throws GitLabApiException {
public Stream<T> stream() throws GitLabApiException, IllegalStateException {

// Make sure that current page is 0, this will ensure the whole list is streamed
// regardless of what page the instance is currently on.
currentPage = 0;
if (pagerStream == null) {
synchronized (this) {
if (pagerStream == null) {

// Create a Stream.Builder to contain all the items. This is more efficient than
// getting a List with all() and streaming that List
Stream.Builder<T> streamBuilder = Stream.builder();
// Make sure that current page is 0, this will ensure the whole list is streamed
// regardless of what page the instance is currently on.
currentPage = 0;

// Iterate through the pages and append each page of items to the stream builder
while (hasNext()) {
next().forEach(streamBuilder);
}
// Create a Stream.Builder to contain all the items. This is more efficient than
// getting a List with all() and streaming that List
Stream.Builder<T> streamBuilder = Stream.builder();

return (streamBuilder.build());
// Iterate through the pages and append each page of items to the stream builder
while (hasNext()) {
next().forEach(streamBuilder);
}

pagerStream = streamBuilder.build();
return (pagerStream);
}
}
}

throw new IllegalStateException("Stream already issued");
}

public Stream<T> lazyStream() {
// Make sure that current page is 0, this will ensure the whole list is streamed
// regardless of what page the instance is currently on.
currentPage = 0;
/**
* Creates a Stream instance for lazily streaming items from the GitLab server.
*
* @return a Stream instance for lazily streaming items from the GitLab server
* @throws IllegalStateException if Stream has already been issued
*/
public Stream<T> lazyStream() throws IllegalStateException {

if (pagerStream == null) {
synchronized (this) {
if (pagerStream == null) {

// Make sure that current page is 0, this will ensure the whole list is streamed
// regardless of what page the instance is currently on.
currentPage = 0;

pagerStream = StreamSupport.stream(new PagerSpliterator<T>(this), false);
return (pagerStream);
}
}
}

return StreamSupport.stream(new PagerSpliterator<T>(this), false);
throw new IllegalStateException("Stream already issued");
}
}
26 changes: 13 additions & 13 deletions src/main/java/org/gitlab4j/api/ProjectApi.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -818,15 +818,15 @@ public Project createProject(Project project, String importUrl) throws GitLabApi
if (isApiVersion(ApiVersion.V3)) {
boolean isPublic = (project.getPublic() != null ? project.getPublic() : project.getVisibility() == Visibility.PUBLIC);
formData.withParam("public", isPublic);

if (project.getTagList() != null && !project.getTagList().isEmpty()) {
throw new IllegalArgumentException("GitLab API v3 does not support tag lists when creating projects");
}
} else {
Visibility visibility = (project.getVisibility() != null ? project.getVisibility() :
project.getPublic() == Boolean.TRUE ? Visibility.PUBLIC : null);
formData.withParam("visibility", visibility);

if (project.getTagList() != null && !project.getTagList().isEmpty()) {
formData.withParam("tag_list", String.join(",", project.getTagList()));
}
Expand Down Expand Up @@ -1057,15 +1057,15 @@ public Project updateProject(Project project) throws GitLabApiException {
formData.withParam("visibility_level", project.getVisibilityLevel());
boolean isPublic = (project.getPublic() != null ? project.getPublic() : project.getVisibility() == Visibility.PUBLIC);
formData.withParam("public", isPublic);

if (project.getTagList() != null && !project.getTagList().isEmpty()) {
throw new IllegalArgumentException("GitLab API v3 does not support tag lists when updating projects");
}
} else {
Visibility visibility = (project.getVisibility() != null ? project.getVisibility() :
project.getPublic() == Boolean.TRUE ? Visibility.PUBLIC : null);
formData.withParam("visibility", visibility);

if (project.getTagList() != null && !project.getTagList().isEmpty()) {
formData.withParam("tag_list", String.join(",", project.getTagList()));
}
Expand All @@ -1090,7 +1090,7 @@ public void deleteProject(Object projectIdOrPath) throws GitLabApiException {

/**
* Forks a project into the user namespace of the authenticated user or the one provided.
* The forking operation for a project is asynchronous and is completed in a background job.
* The forking operation for a project is asynchronous and is completed in a background job.
* The request will return immediately.
*
* <pre><code>POST /projects/:id/fork</code></pre>
Expand All @@ -1109,7 +1109,7 @@ public Project forkProject(Object projectIdOrPath, String namespace) throws GitL

/**
* Forks a project into the user namespace of the authenticated user or the one provided.
* The forking operation for a project is asynchronous and is completed in a background job.
* The forking operation for a project is asynchronous and is completed in a background job.
* The request will return immediately.
*
* <pre><code>POST /projects/:id/fork</code></pre>
Expand All @@ -1130,7 +1130,7 @@ public Project forkProject(Object projectIdOrPath, Integer namespaceId) throws G
* Create a forked from/to relation between existing projects.
*
* <pre><code>POST /projects/:id/fork/:forkFromId</code></pre>
*
*
*
* @param projectIdOrPath projectIdOrPath the project in the form of an Integer(ID), String(path), or Project instance
* @param forkedFromId the ID of the project that was forked from
Expand Down Expand Up @@ -1455,7 +1455,7 @@ public List<ProjectUser> getProjectUsers(Object projectIdOrPath, String search)
}

/**
* Get a Pager of project users matching the specified search string. This Pager includes
* Get a Pager of project users matching the specified search string. This Pager includes
* all project members and all users assigned to project parent groups.
*
* <pre><code>GET /projects/:id/users</code></pre>
Expand Down Expand Up @@ -1644,7 +1644,7 @@ public Optional<ProjectHook> getOptionalHook(Object projectIdOrPath, Integer hoo
* @return the added ProjectHook instance
* @throws GitLabApiException if any exception occurs
*/
public ProjectHook addHook(String projectName, String url, ProjectHook enabledHooks, boolean enableSslVerification, String secretToken)
public ProjectHook addHook(String projectName, String url, ProjectHook enabledHooks, boolean enableSslVerification, String secretToken)
throws GitLabApiException {

if (projectName == null) {
Expand Down Expand Up @@ -2266,9 +2266,9 @@ public void deletePushRules(Object projectIdOrPath) throws GitLabApiException {

/**
* Get a list of projects that were forked from the specified project.
*
*
* <pre><code>GET /projects/:id/forks</code></pre>
*
*
* @param projectIdOrPath projectIdOrPath the project in the form of an Integer(ID), String(path), or Project instance, required
* @return a List of forked projects
* @throws GitLabApiException if any exception occurs
Expand Down Expand Up @@ -2382,7 +2382,7 @@ public Project transferProject(Object projectIdOrPath, String namespace) throws
}

/**
* Uploads and sets the project avatar for the specified project.
* Uploads and sets the project avatar for the specified project.
*
* <pre><code>PUT /projects/:id/uploads</code></pre>
*
Expand All @@ -2395,4 +2395,4 @@ public Project setProjectAvatar(Object projectIdOrPath, File avatarFile) throws
Response response = putUpload(Response.Status.OK, "avatar", avatarFile, "projects", getProjectIdOrPath(projectIdOrPath));
return (response.readEntity(Project.class));
}
}
}
Loading