fluent-bit output plugin for google pubsub

This plugin is used to publish data to queue in google pubsub.

You could easily use it.

Build

A bin directory already has been made binaries for mac, linux.

If you should directly make binaries for mac, linux

# local machine binary
$ bash make.sh build

# Your machine is mac, and if you should do to retry cross compiling for linux.
# A command in below is required a docker.  
$ bash make.sh build_linux

# qanda-for-fluent-bit
$ docker build -f docker/Dockerfile . -t qanda-for-fluent-bit:beta

Usage

configuration options for fluent-bit.conf

Key	Description	Default
Project	google cloud project id	NONE(required)
Topic	google pubsub topic name	NONE(required)
JwtPath	jwt file path for accessible google cloud project	NONE(required)
Debug	print debug log	false(optional)
Timeout	the maximum time that the client will attempt to publish a bundle of messages. (millsecond)	60000 (optional)
DelayThreshold	publish a non-empty batch after this delay has passed. (millsecond)	1
ByteThreshold	publish a batch when its size in bytes reaches this value.	1000000
CountThreshold	publish a batch when it has been reached count of messages.	100

Example fluent-bit.conf

[Output]
    Name pubsub
    Match *
    Project your-project(custom)
    Topic your-topic-name(custom)
    Jwtpath your-jwtpath(custom)    

Example exec

$ fluent-bit -c [your config file] -e pubsub.so 

GCP Cloudbuild 배포

1. 이미지 배포는 GCP CloudBuild를 통해서 배포하고 Cloudbuild Trigger는 다음과 같습니다. GCP CloudBduil Trigger

2. 빌드 후 Artifact-registry 저장됩니다. Fluent-Bit 이미지

3. Melon Fluent-Bit 배포 예시 yaml 정보 버전 upgrade 시 tag 정보로 업그레이드 합니다.

        fluentbit:
          enabled: true
          tag: 3.2.10-16ae37a
          mountPaths:
          - /tmp/data-pipeline
          infos:
          - logPath: /tmp/data-pipeline/data-pipeline.log
            pubsub_topic: qanda_log
            project_id: qp-data-serverlog-0a34
          resources:
            requests:
              cpu: 10m
              memory: 128Mi
            limits:
              memory: 256Mi