How to Convert Arrays of Hashes Into a Structured Key-Value Format During Log Processing

Posted on

In some log formats, fields can be arrays of hashes, requiring conversion into a structured key-value format. Fluentd supports this through inline Ruby scripts, enabling transformations during log processing.

For example, I need to convert the event_data field:

{
  "event_data": [
    {"name": "feature_enabled", "boolValue": true},
    {"name": "user_roles", "multiValue": ["admin", "editor"]},
    {"name": "session_id", "value": "abc123"}
  ]
}

Step 1. Write Ruby-Based Transformation Logic

The transformation requires iterating over the event_data array, extracting meaningful information, and producing a hash. The logic can be implemented as:

record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h
How it works:
  • Access the event_data Field:
          record['event_data'] retrieves the array of hashes.
  • Iterate Over the Array:
          .map iterates over each item in the array, returning a transformed version.
  • Extract Key-Value Pairs:
          [item['name'], item['boolValue'] || item['multiValue'] || item['value']]:
         item['name']: The key in the resulting hash.
         item['boolValue'] || item[‘multiValue’] || item[‘value’]: The first non-nil value among the fields.
  • Convert to a Hash:
          .to_h transforms the array of key-value pairs into a hash.If event_data is missing, the original value (nil) is retained. `So to make sure that the field exists, before applying the transformation, I use !record.dig('event_data').nil?  to skip the empty field.
You will see it at step 2.

Step 2. Setup Implementing in Fluentd

Use the record_transformer filter to apply the transformation dynamically. Enable Ruby scripting with the parameter enable_ruby true.

In Fluentd Configuration:

<filter app.logs>
  @type record_transformer
  enable_ruby true
  <record>
    event_data ${!record.dig('event_data').nil? ? record['event_data'].map { |item| [item['name'], item['boolValue'] || item['multiValue'] || item['value']] }.to_h : record['event_data']}
  </record>
</filter>
Key Components:
  • Conditional Logic:
       !record.dig('event_data').nil?: Ensures the field exists before applying the transformation.
  • Transformation Logic:
       Converts event_data into a normalized hash using Ruby.
  • Fallback:
       If event_data is missing, the original value (nil) is retained.
For example, I received this output:
{
  "event_data": {
    "feature_enabled": true,
    "user_roles": ["admin", "editor"],
    "session_id": "abc123"
  }
}

The post How to Convert Arrays of Hashes Into a Structured Key-Value Format During Log Processing appeared first on SOC Prime.

Related Posts