Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/src/main/asciidoc/configuration.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -517,6 +517,8 @@ NOTE: When using the Tika `ParserBolt` alongside `JSoupParserBolt`, set `jsoup.t
Integration with AWS services: CloudSearch for indexing and S3 for content caching.
See the link:https://github.com/apache/stormcrawler/tree/main/external/aws[aws module] for full details.

NOTE: AWS CloudSearch support (the `cloudsearch.*` settings below) is deprecated and will be removed in the next major release of StormCrawler.

[cols="1,1,3", options="header"]
|===
| key | default value | description
Expand Down
2 changes: 1 addition & 1 deletion docs/src/main/asciidoc/internals.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ The purpose of crawlers is often to index web pages to make them searchable. The

* link:https://github.com/apache/stormcrawler/blob/main/external/solr/src/main/java/org/apache/stormcrawler/solr/bolt/IndexerBolt.java[Apache SOLR]
* link:https://github.com/apache/stormcrawler/blob/main/external/opensearch/src/main/java/org/apache/stormcrawler/opensearch/bolt/IndexerBolt.java[OpenSearch]
* link:https://github.com/apache/stormcrawler/blob/main/external/aws/src/main/java/org/apache/stormcrawler/aws/bolt/CloudSearchIndexerBolt.java[AWS CloudSearch]
* link:https://github.com/apache/stormcrawler/blob/main/external/aws/src/main/java/org/apache/stormcrawler/aws/bolt/CloudSearchIndexerBolt.java[AWS CloudSearch] (deprecated, will be removed in the next major release)

All of these extend the class link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org/apache/stormcrawler/indexing/AbstractIndexerBolt.java[AbstractIndexerBolt].

Expand Down
4 changes: 3 additions & 1 deletion external/aws/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# stormcrawler-aws
================================

AWS resources for StormCrawler, currently contains an indexer bolt for [CloudSearch](https://aws.amazon.com/cloudsearch/) and another bolt for storing and retrieving web pages to/from [S3](https://aws.amazon.com/s3/).
AWS resources for StormCrawler, currently contains an indexer bolt for [CloudSearch](https://aws.amazon.com/cloudsearch/) (**deprecated**, see below) and another bolt for storing and retrieving web pages to/from [S3](https://aws.amazon.com/s3/).

This module is built on the [AWS SDK for Java 2.x](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html). Credentials are resolved through the SDK's [default credentials provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) (environment variables, Java system properties, `~/.aws/credentials`, container/instance profiles, etc.).

Expand All @@ -21,6 +21,8 @@ Edit `~/.aws/credentials`, see [http://docs.aws.amazon.com/cli/latest/userguide/

## CloudSearch

> **Deprecated:** AWS CloudSearch support is deprecated and will be removed in the next major release of StormCrawler.

* How to use?

Add stormcrawler-aws as a Maven dependency, use the class CloudSearchIndexWriter in your Storm topology alongside the core StormCrawler components and create a yaml configuration file (see below).
Expand Down
3 changes: 3 additions & 0 deletions external/aws/aws-conf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
# CloudSearch Configuration #
###############################

# NOTE: AWS CloudSearch support is deprecated and will be removed
# in the next major release of StormCrawler.

# max number of docs to keep in buffer
# before sending to CS irrespective of size limits
cloudsearch.batch.maxSize: 100
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@

package org.apache.stormcrawler.aws.bolt;

/**
* @deprecated AWS CloudSearch support is deprecated and will be removed in the next major release
* of StormCrawler.
*/
@Deprecated
public interface CloudSearchConstants {
public static final String CLOUDSEARCH_PREFIX = "cloudsearch.";
public static final String ENDPOINT = CLOUDSEARCH_PREFIX + "endpoint";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,13 @@
import software.amazon.awssdk.services.cloudsearchdomain.model.UploadDocumentsRequest;
import software.amazon.awssdk.services.cloudsearchdomain.model.UploadDocumentsResponse;

/** Writes documents to CloudSearch. */
/**
* Writes documents to CloudSearch.
*
* @deprecated AWS CloudSearch support is deprecated and will be removed in the next major release
* of StormCrawler.
*/
@Deprecated
public class CloudSearchIndexerBolt extends AbstractIndexerBolt {

public static final Logger LOG = LoggerFactory.getLogger(CloudSearchIndexerBolt.class);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@
import java.util.regex.Pattern;
import org.apache.commons.codec.digest.DigestUtils;

/**
* @deprecated AWS CloudSearch support is deprecated and will be removed in the next major release
* of StormCrawler.
*/
@Deprecated
public class CloudSearchUtils {

private static final Pattern INVALID_XML_CHARS =
Expand Down
Loading