Skip to content

opensearch

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

opensearch

README.md

Ingesting data into a domain using Amazon OpenSearch Ingestion

osis-domain-pipeline

This is an Amazon OpenSearch ingestion project for CDK development with Python.

This project builds on the following tutorial: Ingesting data into a domain using Amazon OpenSearch Ingestion.

This project shows you how to use Amazon OpenSearch Ingestion to configure a simple pipeline and ingest data into an Amazon OpenSearch.

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

(.venv) $ pip install -r requirements.txt

At this point you can now synthesize the CloudFormation template for this code.

(.venv) $ export CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
(.venv) $ export CDK_DEFAULT_REGION=$(aws configure get region)
(.venv) $ cdk synth --all

To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command.

Deploy

Use cdk deploy command to create the stack shown above.

Step 1: Create an OpenSearch Domain

(.venv) $ cdk deploy --require-approval never \
                     OpsDomainVpc \
                     OpsDomainStack \
                     OpsClientEC2Stack

Step 2: Create an OpenSearch Ingestion Pipeline Role

(.venv) $ cdk deploy --require-approval never OpsDomainPipelineRoleStack

The opensearch domain has been provisioned with fine-grained access control for authentication.

Therefore, you need to take extra steps to provide the pipeline access to an OpenSearch domain.

The domain uses a master user in the internal user database and HTTP basic authentication for OpenSearch Dashboards. So, you can't pass the master username and password directly into the pipeline configuration. Instead, you need to map the pipeline role (sts_role_arn) to the OpenSearch all_access backend role.

In order to do that, complete the following steps:

  1. The Amazon OpenSearch cluster is provisioned in a VPC. Hence, the Amazon OpenSearch endpoint and dashboard are not available over the internet. In order to access the endpoints, we have to create a ssh tunnel and do local port forwarding.

    1. Install EC2 Instance Connect CLI 
      sudo pip install ec2instanceconnectcli
    2. Run 
      mssh --region {region} ec2-user@{bastion-ec2-instance-id} -N -L 9200:{opensearch-endpoint}:443
    • ex) 
      $ mssh --region us-east-1 ec2-user@i-0203f0d6f37ccbe5b -N -L 9200:vpc-retail-qvwlxanar255vswqna37p2l2cy.us-east-1.es.amazonaws.com:443

  2. Connect to https://localhost:9200/_dashboards/app/login? in a web browser.

  3. Enter the master user and password that you set up when you created the Amazon OpenSearch Service endpoint. The user and password is stored in the AWS Secrets Manager as a name such as OpenSearchMasterUserSecret1-xxxxxxxxxxxx. opensearch-secrets

  4. In the Welcome screen, click the toolbar icon to the left side of Home button. Choose Security. ops-dashboards-sidebar-menu-security

  5. Choose Security, Roles, and then map the ingestion pipeline role to the all_access roles. opensearch_additional_master_users

Step 3: Create an OpenSearch Ingestion Pipeline

(.venv) $ cdk deploy OpsDomainIngestionStack

Clean Up

Delete the CloudFormation stack by running the below command.

(.venv) $ cdk destroy --force --all

Useful commands

  • cdk ls list all stacks in the app
  • cdk synth emits the synthesized CloudFormation template
  • cdk deploy deploy this stack to your default AWS account/region
  • cdk diff compare deployed stack with current state
  • cdk docs open CDK documentation

Enjoy!

Run Test

Step 1: Ingest some sample data

First, get the ingestion URL from the Pipeline settings page:

osis-pipeline-settings

Then, ingest some sample data. The following sample request uses awscurl to send a single log file to the my_logs index:

$ awscurl --service osis --region us-east-1 \
     -X POST \
     -H "Content-Type: application/json" \
     -d '[{"time":"2014-08-11T11:40:13+00:00","remote_addr":"122.226.223.69","status":"404","req
   uest":"GET http://www.k2proxy.com//hello.html HTTP/1.1","http_user_agent":"Mozilla/4.0 (compatible; WOW64; SLCC2;)"}]' \
   https://{pipeline-endpoint}.us-east-1.osis.amazonaws.com/log-pipeline/test_ingestion_path

You should see a 200 OK response.

Step 2: Query the sample data

Now, query the application_logs index to ensure that the log entry was successfully ingested.

In OpenSearch Dashboards, choose Dev Tools, and run the following query: ops-dev-tools

GET application_logs/_search
{
  "query": {
    "match_all": {}
  }
}

Sample response:

{
  "took": 21,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "application_logs",
        "_id": "lyzBVYgBPgOW541j2YpC",
        "_score": 1,
        "_source": {
          "time": "2014-08-11T11:40:13+00:00",
          "remote_addr": "122.226.223.69",
          "status": "404",
          "request": "GET http://www.k2proxy.com//hello.html HTTP/1.1",
          "http_user_agent": "Mozilla/4.0 (compatible; WOW64; SLCC2;)",
          "@timestamp": "2023-05-24T07:16:29.708Z"
        }
      }
    ]
  }
}

References