これは、なにをしたくて書いたもの？
ElasticMQを使っていきたいと思うのですが、リソース定義をTerraformで行えないかなと思いまして。
結果を見ると、できるにはできるのですがちょっと難ありです。
ElasticMQ
ElasticMQは、Amazon SQS互換のインターフェースを実装したメッセージキューです。ScalaとAkkaを使って実装されています。
以前にも使ったことがあります。
Amazon SQS互換のElasticMQを使って、Temoporary Queue＋RPCを試してみる - CLOVER🍀
この時はAWS CLIでキューを作成したのですが、できればTerraformで行いたいなと。
Terraform AWS Providerとカスタムエンドポイント
TerraformでAWS互換のサービスを利用するには、カスタムエンドポイントを設定します。Amazon DynamoDB LocalとLocalStackを使った例が、
TerraformのAWS Providerに書かれていますね。
Custom Service Endpoint Configuration / Connecting to Local AWS Compatible Solutions
ElasticMQを使う時も、こちらに習ってみたいと思います。
環境
今回の環境は、こちら。
$ java --version openjdk 17.0.4 2022-07-19 OpenJDK Runtime Environment (build 17.0.4+8-Ubuntu-120.04) OpenJDK 64-Bit Server VM (build 17.0.4+8-Ubuntu-120.04, mixed mode, sharing) $ terraform version Terraform v1.3.1 on linux_amd64 $ aws --version aws-cli/2.8.0 Python/3.9.11 Linux/5.4.0-126-generic exe/x86_64.ubuntu.20 prompt/off
ElasticMQをインストールする
まずは、ElasticMQをインストールします。こちらは簡単で、JARファイルをダウンロードして
$ curl -LO https://s3-eu-west-1.amazonaws.com/softwaremill-public/elasticmq-server-1.3.11.jar
java -jarで起動。
$ java -jar elasticmq-server-1.3.11.jar
以下のようなログを出力しつつ、起動します。
20:52:41.650 [main] INFO org.elasticmq.server.Main$ - Starting ElasticMQ server (1.3.11) ... 20:52:42.104 [elasticmq-akka.actor.default-dispatcher-4] INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 20:52:42.843 [elasticmq-akka.actor.default-dispatcher-4] INFO o.e.rest.sqs.TheSQSRestServerBuilder - Started SQS rest server, bind address 0.0.0.0:9324, visible server address http://localhost:9324 20:52:42.909 [main] INFO o.e.rest.sqs.TheSQSRestServerBuilder - Metrics MBean org.elasticmq:name=Queues successfully registered 20:52:42.949 [elasticmq-akka.actor.default-dispatcher-4] INFO o.e.r.s.TheStatisticsRestServerBuilder - Started statistics rest server, bind address 0.0.0.0:9325 20:52:42.953 [main] INFO org.elasticmq.server.Main$ - === ElasticMQ server (1.3.11) started in 1587 ms ===
ポートは9324と9325を使いますが、キューへのアクセスに使用するのは9324ポートです。
ちなみに、今回は使いませんが、アプリケーションに組み込んだりDockerコンテナで起動することもできます。
- ElasticMQ / Starting an embedded ElasticMQ server with an SQS interface
- ElasticMQ / ElasticMQ via Docker
- ElasticMQ / ElasticMQ via Docker (full JVM)
TerraformでElasticMQにキューを作成する
では、TerraformでElasticMQにキューを作成しましょう。
Terraformのリソース定義ファイルは、こんな感じで作成。
main.tf
terraform { required_version = "1.3.1" required_providers { aws = { source = "hashicorp/aws" version = "4.33.0" } } } provider "aws" { access_key = "mock_access_key" region = "us-east-1" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true endpoints { sqs = "http://localhost:9324" } } resource "aws_sqs_queue" "queue" { name = "my-queue" } resource "aws_sqs_queue" "queue_fifo" { name = "my-queue.fifo" fifo_queue = true content_based_deduplication = true } output "queue_url" { value = aws_sqs_queue.queue.url } output "queue_fifo_url" { value = aws_sqs_queue.queue_fifo.url }
通常のキューとFIFOキューを作成します。
ポイントは、AWS Providerの設定ですね。
provider "aws" { access_key = "mock_access_key" region = "us-east-1" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true endpoints { sqs = "http://localhost:9324" } }
Amazon DynamoDB LocalやLocalStackの時と同じように、ダミーのクレデンシャルやエンドポイントを指定します。
Custom Service Endpoint Configuration / Connecting to Local AWS Compatible Solutions
あとはinitして
$ terraform init
planで確認。
$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_sqs_queue.queue will be created
+ resource "aws_sqs_queue" "queue" {
+ arn = (known after apply)
+ content_based_deduplication = false
+ deduplication_scope = (known after apply)
+ delay_seconds = 0
+ fifo_queue = false
+ fifo_throughput_limit = (known after apply)
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "my-queue"
+ name_prefix = (known after apply)
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ redrive_allow_policy = (known after apply)
+ redrive_policy = (known after apply)
+ tags_all = (known after apply)
+ url = (known after apply)
+ visibility_timeout_seconds = 30
}
# aws_sqs_queue.queue_fifo will be created
+ resource "aws_sqs_queue" "queue_fifo" {
+ arn = (known after apply)
+ content_based_deduplication = true
+ deduplication_scope = (known after apply)
+ delay_seconds = 0
+ fifo_queue = true
+ fifo_throughput_limit = (known after apply)
+ id = (known after apply)
+ kms_data_key_reuse_period_seconds = (known after apply)
+ max_message_size = 262144
+ message_retention_seconds = 345600
+ name = "my-queue.fifo"
+ name_prefix = (known after apply)
+ policy = (known after apply)
+ receive_wait_time_seconds = 0
+ redrive_allow_policy = (known after apply)
+ redrive_policy = (known after apply)
+ tags_all = (known after apply)
+ url = (known after apply)
+ visibility_timeout_seconds = 30
}
Plan: 2 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ queue_fifo_url = (known after apply)
+ queue_url = (known after apply)
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
うまくいきそうな感じなので、apply。
$ terraform apply
なのですが、キューの作成が終わらずタイムアウトします。
Changes to Outputs:
+ queue_fifo_url = (known after apply)
+ queue_url = (known after apply)
aws_sqs_queue.queue_fifo: Creating...
aws_sqs_queue.queue: Creating...
aws_sqs_queue.queue_fifo: Still creating... [10s elapsed]
aws_sqs_queue.queue: Still creating... [10s elapsed]
aws_sqs_queue.queue: Still creating... [20s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [20s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [30s elapsed]
aws_sqs_queue.queue: Still creating... [30s elapsed]
aws_sqs_queue.queue: Still creating... [40s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [40s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [50s elapsed]
aws_sqs_queue.queue: Still creating... [50s elapsed]
aws_sqs_queue.queue: Still creating... [1m0s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m0s elapsed]
aws_sqs_queue.queue: Still creating... [1m10s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m10s elapsed]
aws_sqs_queue.queue: Still creating... [1m20s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m20s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m30s elapsed]
aws_sqs_queue.queue: Still creating... [1m30s elapsed]
aws_sqs_queue.queue: Still creating... [1m40s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m40s elapsed]
aws_sqs_queue.queue: Still creating... [1m50s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m50s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [2m0s elapsed]
aws_sqs_queue.queue: Still creating... [2m0s elapsed]
╷
│ Error: waiting for SQS Queue (http://localhost:9324/000000000000/my-queue) attributes create: timeout while waiting for state to become 'equal' (last state: 'notequal', timeout: 2m0s)
│
│ with aws_sqs_queue.queue,
│ on main.tf line 25, in resource "aws_sqs_queue" "queue":
│ 25: resource "aws_sqs_queue" "queue" {
│
╵
╷
│ Error: waiting for SQS Queue (http://localhost:9324/000000000000/my-queue.fifo) attributes create: timeout while waiting for state to become 'equal' (last state: 'notequal', timeout: 2m0s)
│
│ with aws_sqs_queue.queue_fifo,
│ on main.tf line 29, in resource "aws_sqs_queue" "queue_fifo":
│ 29: resource "aws_sqs_queue" "queue_fifo" {
│
╵
outputも空です。
$ terraform output ╷ │ Warning: No outputs found │ │ The state file either has no outputs defined, or all the defined outputs are empty. Please define an output in your configuration with the `output` keyword and run │ `terraform refresh` for it to become available. If you are using interpolation, please verify the interpolated value is not empty. You can use the `terraform console` │ command to assist. ╵
なお、destroyしようとすると
$ terraform destroy
2つのリソースの破棄を聞かれます。
aws_sqs_queue.queue_fifo: Refreshing state... [id=http://localhost:9324/000000000000/my-queue.fifo]
aws_sqs_queue.queue: Refreshing state... [id=http://localhost:9324/000000000000/my-queue]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
# aws_sqs_queue.queue will be destroyed
- resource "aws_sqs_queue" "queue" {
- arn = "arn:aws:sqs:elasticmq:000000000000:my-queue" -> null
- content_based_deduplication = false -> null
- delay_seconds = 0 -> null
- fifo_queue = false -> null
- id = "http://localhost:9324/000000000000/my-queue" -> null
- kms_data_key_reuse_period_seconds = 300 -> null
- max_message_size = 0 -> null
- message_retention_seconds = 0 -> null
- name = "my-queue" -> null
- receive_wait_time_seconds = 0 -> null
- sqs_managed_sse_enabled = false -> null
- tags = {} -> null
- tags_all = {} -> null
- url = "http://localhost:9324/000000000000/my-queue" -> null
- visibility_timeout_seconds = 30 -> null
}
# aws_sqs_queue.queue_fifo will be destroyed
- resource "aws_sqs_queue" "queue_fifo" {
- arn = "arn:aws:sqs:elasticmq:000000000000:my-queue.fifo" -> null
- content_based_deduplication = true -> null
- delay_seconds = 0 -> null
- fifo_queue = true -> null
- id = "http://localhost:9324/000000000000/my-queue.fifo" -> null
- kms_data_key_reuse_period_seconds = 300 -> null
- max_message_size = 0 -> null
- message_retention_seconds = 0 -> null
- name = "my-queue.fifo" -> null
- receive_wait_time_seconds = 0 -> null
- sqs_managed_sse_enabled = false -> null
- tags = {} -> null
- tags_all = {} -> null
- url = "http://localhost:9324/000000000000/my-queue.fifo" -> null
- visibility_timeout_seconds = 30 -> null
}
Plan: 0 to add, 0 to change, 2 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value:
ここで「yes」とすると、キューが破棄されます。
というわけで、タイムアウトはしているのですが、実はキューはできています。
このようなエラーメッセージが出るまで待たずに途中でCtrl-cで止めたりしても、裏でキューはできていました。
〜省略〜
aws_sqs_queue.queue: Still creating... [1m40s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m40s elapsed]
aws_sqs_queue.queue: Still creating... [1m50s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [1m50s elapsed]
aws_sqs_queue.queue_fifo: Still creating... [2m0s elapsed]
aws_sqs_queue.queue: Still creating... [2m0s elapsed]
╷
│ Error: waiting for SQS Queue (http://localhost:9324/000000000000/my-queue) attributes create: timeout while waiting for state to become 'equal' (last state: 'notequal', timeout: 2m0s)
│
│ with aws_sqs_queue.queue,
│ on main.tf line 25, in resource "aws_sqs_queue" "queue":
│ 25: resource "aws_sqs_queue" "queue" {
│
╵
╷
│ Error: waiting for SQS Queue (http://localhost:9324/000000000000/my-queue.fifo) attributes create: timeout while waiting for state to become 'equal' (last state: 'notequal', timeout: 2m0s)
│
│ with aws_sqs_queue.queue_fifo,
│ on main.tf line 29, in resource "aws_sqs_queue" "queue_fifo":
│ 29: resource "aws_sqs_queue" "queue_fifo" {
│
╵
作成自体は、10秒と経たずに完了するようです。
確認する
$ export AWS_ACCESS_KEY_ID=mock_access_key $ export AWS_SECRET_ACCESS_KEY=mock_secret_key $ export AWS_DEFAULT_REGION=us-east-1
キューの一覧。
$ aws --endpoint-url http://localhost:9324 sqs list-queues
{
"QueueUrls": [
"http://localhost:9324/000000000000/my-queue",
"http://localhost:9324/000000000000/my-queue.fifo"
]
}
キューがありますね。
キューの属性を確認してみます。
通常のキュー。
$ aws --endpoint-url http://localhost:9324 sqs get-queue-attributes --queue-url http://localhost:9324/000000000000/my-queue --attribute-names All
{
"Attributes": {
"VisibilityTimeout": "30",
"DelaySeconds": "0",
"ReceiveMessageWaitTimeSeconds": "0",
"ApproximateNumberOfMessages": "0",
"ApproximateNumberOfMessagesNotVisible": "0",
"ApproximateNumberOfMessagesDelayed": "0",
"CreatedTimestamp": "1664712029",
"LastModifiedTimestamp": "1664712029",
"QueueArn": "arn:aws:sqs:elasticmq:000000000000:my-queue"
}
}
FIFOキュー。
$ aws --endpoint-url http://localhost:9324 sqs get-queue-attributes --queue-url http://localhost:9324/000000000000/my-queue.fifo --attribute-names All
{
"Attributes": {
"VisibilityTimeout": "30",
"DelaySeconds": "0",
"ReceiveMessageWaitTimeSeconds": "0",
"ApproximateNumberOfMessages": "0",
"ApproximateNumberOfMessagesNotVisible": "0",
"ApproximateNumberOfMessagesDelayed": "0",
"CreatedTimestamp": "1664712029",
"LastModifiedTimestamp": "1664712029",
"QueueArn": "arn:aws:sqs:elasticmq:000000000000:my-queue.fifo",
"ContentBasedDeduplication": "true",
"FifoQueue": "true"
}
}
Terraformのリソース定義に指定した値も、ちゃんと反映されていそうですね。
メッセージの送受信を試してみましょう。
通常のキューにメッセージを送信。
$ aws --endpoint-url http://localhost:9324 sqs send-message --queue-url http://localhost:9324/000000000000/my-queue --message-body 'Hello Normal Queue'
{
"MD5OfMessageBody": "56f5b1a443d080fe35c21bf08ba1c193",
"MessageId": "41b785a6-2807-455a-b633-08250b29df66"
}
受信。
$ aws --endpoint-url http://localhost:9324 sqs receive-message --queue-url http://localhost:9324/000000000000/my-queue
{
"Messages": [
{
"MessageId": "41b785a6-2807-455a-b633-08250b29df66",
"ReceiptHandle": "41b785a6-2807-455a-b633-08250b29df66#64946d96-fae2-455f-83a8-3cb5ad620f79",
"MD5OfBody": "56f5b1a443d080fe35c21bf08ba1c193",
"Body": "Hello Normal Queue"
}
]
}
OKですね。
FIFOキュー。
メッセージの送信。
$ aws --endpoint-url http://localhost:9324 sqs send-message --queue-url http://localhost:9324/000000000000/my-queue.fifo --message-group-id 1 --message-body 'Hello FIFO Queue'
{
"MD5OfMessageBody": "5aea1fee988fbe6da0b76d02bdbc147a",
"MessageId": "48cad0b6-90cd-401f-a55e-becabf43b68f",
"SequenceNumber": "0"
}
受信。
$ aws --endpoint-url http://localhost:9324 sqs receive-message --queue-url http://localhost:9324/000000000000/my-queue.fifo
{
"Messages": [
{
"MessageId": "48cad0b6-90cd-401f-a55e-becabf43b68f",
"ReceiptHandle": "48cad0b6-90cd-401f-a55e-becabf43b68f#eb63369f-a030-4ed6-9368-084548958bdc",
"MD5OfBody": "5aea1fee988fbe6da0b76d02bdbc147a",
"Body": "Hello FIFO Queue"
}
]
}
動作自体はOKですね。タイムアウトするのが気持ち悪いですが…。
ちなみに、この問題はAWS ProviderやLocalStackでも挙がっていることがあるので、なにか相性があるのかも…。
まとめ
ElasticMQのキュー定義をTerraformを使って行ってみました。
タイムアウトするのが難点ですが、動きはするのでまあいいかなと。TerraformとAWS Provider、ElasticMQのバージョンの組み合わせによっては
うまく動くのかもしれませんが。
気になる場合は、LocalStackで使った場合はこうならなかったのでこちらに切り替えたり、
LocalStackでAmazon SQSのFIFOキューを試してみる（AWS SDK for Javaを使用） - CLOVER🍀
そもそもTerraformで構築するのではなくてElasticMQのやり方でキューを作成するのがよいのでしょう。
設定ファイルで定義することもできるようですし。
