RDSC-3621 format epoch to date string

content/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values.md

@@ -0,0 +1,189 @@
+---
+Title: Formatting date and time values
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Formatting date and time values
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+  real time.
+type: integration
+weight: 40
+---
+
+How to format date and time values depends on the source database and the data type of the field, and how they are represented in the incoming record. Below are examples for different databases and data types.
+
+## Oracle
+
+Oracle supports the following date and time data types:
+
+- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch
+  ```yaml
+  transform:
+  - uses: add_field
+    with:
+      fields:
+        - field: formatted_date
+          language: sql
+          # Date is stored as a Unix timestamp in milliseconds so you need to
+          # divide it by 1000 to convert it to seconds.
+          expression: STRFTIME('%Y-%m-%d %H:%M:%S', DATE / 1000, 'unixepoch')
+          # Example: 1749047572000 is transformed to 2025-06-04 14:32:52
+  ```
+- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
+You can format it similarly to `DATE`, but you need to divide the value by the appropriate factor based on the precision.
+
+- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information.
+
+- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information.
+
+  Both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE` are supported by SQLite and can be formatted using the `STRFTIME` function.
+
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: seconds_since_epoch
+            language: sql
+            # Convert the timestamp with local time zone to seconds since epoch.
+            expression: STRFTIME('%s', TIMESTAMP_FIELD)
+
+          - field: date_from_timestamp
+            language: sql
+            # Convert the timestamp with local time zone to date and time.
+            expression: STRFTIME('%Y-%m-%d %H:%M:%S', TIMESTAMP_FIELD)
+  ```
+
+----
+
+## SQL Server
+SQL Server supports the following date and time data types:
+
+- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: with_default_date_format
+            language: sql
+            # Uses the default DATE format
+            expression: DATE(event_date * 86400, 'unixepoch')
+
+          - field: with_custom_date_format
+            language: sql
+            # Uses the default DATE format
+            expression: STRFTIME('%Y/%m/%d', event_date * 86400, 'unixepoch')
+  ```
+
+- `datetime`, `smalldatetime` - represented by Debezium as number of milliseconds since epoch. You have to divide the value by 1000 to convert it to seconds since epoch and then use the `STRFTIME` function to format it.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_datetime
+            language: sql
+            expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetime / 1000, 'unixepoch')
+  ```
+
+- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
+
+- `time` - the time of milliseconds since midnight.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_time
+            language: sql
+            expression: TIME(event_time, 'unixepoch', 'utc')
+  ```
+
+- `datetimeoffset` - represented as a timestamp with timezone information e.g. `2025-05-27T15:21:42.864Z` and `2025-01-02T14:45:30.123+05:00`. 
+  ```yaml
+  transform:
+  - uses: add_field
+    with:
+      fields:
+        - field: formatted_datetimeoffset
+          language: sql
+          expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_datetimeoffset)
+  ```
+
+
+
+
+<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->
+
+
+
+----
+
+## PostgreSQL
+
+PostgreSQL supports the following date and time data types:
+
+- `date` - represented by Debezium as number of days since epoch (1970-01-01). You can multiply the value by 86400 (the number of seconds in a day) to convert it to seconds since epoch and then use the `STRFTIME` or `DATE` functions to format it.
+  ```yaml
+    transform:
+        - uses: add_field
+          with:
+            fields:
+              - field: with_default_date_format
+                language: sql
+                # Uses the default DATE format
+                expression: DATE(event_date * 86400, 'unixepoch')
+  ```
+
+- `time` - the time of microseconds since midnight.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_time
+            language: sql
+            # Divide by 1000000 to convert microseconds to seconds
+            expression: TIME(event_time / 1000000, 'unixepoch', 'utc')
+  ```
+
+- `time with time zone` - a string representation of the time with timezone information, where the timezone is GMT, example `07:15:00Z`.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_time_with_tz
+            language: sql
+            expression: STRFTIME('%H:%M:%S', event_time_with_time_zone)
+  ```
+
+- `timestamp` - represented by Debezium as a 64-bit integer representing the microseconds since epoch. You can use the `STRFTIME` function to format it.
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_timestamp
+            language: sql
+            # Divide by 1000000 to convert microseconds to seconds
+            expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp / 1000000, 'unixepoch')
+  ```
+
+- `timestamp with time zone` - represented by Debezium as a string representation of the timestamp with time zone information, where the timezone is GMT, e.g. `2025-06-07T10:15:00.000000Z`
+  ```yaml
+  transform:
+    - uses: add_field
+      with:
+        fields:
+          - field: formatted_timestamp_with_tz
+            language: sql
+            # Divide by 1000000 to convert microseconds to seconds
+            expression: STRFTIME('%Y-%m-%d %H:%M:%S', event_timestamp_with_time_zone)
+  ```

content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md

@@ -167,4 +167,4 @@ like the following:
 6) "3:35.0"
 7) "storagesize"
 8) "6.71MB"
-```
+```

content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-expiration-example.md

@@ -53,112 +53,34 @@ output:
 
 In some cases, you can also set the expiration time based on a field that contains a date, datetime, or timestamp value, but it depends on the source database and the data types it supports. See the examples below for your specific source database and data type.
 
-### Oracle examples
+There are two main approaches you can use to set the expiration time based on a date, datetime, or timestamp field:
 
-The transformation depends on the data type of the field in the source database:
+- For values representing a passed amount of time e.g. milliseconds since epoch start, you have to convert the value to seconds since epoch and subtracting the current time in seconds since epoch from it.
 
-- `DATE` - represented by debezium as a 64-bit integer representing the milliseconds since epoch
-  ```yaml
-  output:
-    - uses: redis.write
-      with:
-        data_type: hash
-        expire:
-          # To set the expiration time to a date field, convert the value to seconds and subtract the current time in seconds since epoch
-          expression: EXPIRES_DATE / 1000 - STRFTIME('%s', 'now')
-          language: sql
-  ```
-- `TIMESTAMP` - the value is represented by Debezium as a 64-bit integer and depends on the number of decimal places of precision of the column, representing fractions of a second. For example, if the column is defined as `TIMESTAMP(6)`, there are six decimal places and so the value is represented as microseconds since epoch (since there are 10^6 microseconds in each second).
-  ```yaml
-    output:
-      - uses: redis.write
-        with:
-          data_type: hash
-          expire:
-            # To set the expiration time to a date field, convert the value to seconds (divider differs based on the fractional second precision) and subtract the current time in seconds since epoch. Example below is for 6 digits of precision.
-            expression: EXPIRES_TIMESTAMP / 1000000 - STRFTIME('%s', 'now')
-            language: sql
-  ```
-- `TIMESTAMP WITH TIME ZONE` - the value is represented as string representation of the timestamp with time zone information.
-- `TIMESTAMP WITH LOCAL TIME ZONE` - the value is represented as string representation of the timestamp with local time zone information.
-
-  For both `TIMESTAMP WITH TIME ZONE` and `TIMESTAMP WITH LOCAL TIME ZONE`, a two-step approach is needed. First, calculate the difference between the given time and now in seconds and then invert the value. 
     ```yaml
-      transform:
-        - uses: add_field
-          with:
-            fields:
-              - field: expire_seconds
-                language: jmespath
-                expression: time_delta_seconds(EXPIRES_TS_TZ)
-      output:
+    output:
         - uses: redis.write
           with:
             data_type: hash
             expire:
-              # `time_delta_seconds` Returns the number of seconds between a given dt and now. 
-              # A negative value means that the given dt is in the future, so we need to invert it.
-              # A positive value means that the given dt is in the past, so set the expiration to -1 (expire immediately).
-              expression: CASE WHEN expire_seconds < 0 THEN -expire_seconds ELSE -1 END
+              # To set the expiration time to a date field, convert the value to
+              # seconds (e.g. divide it by 1000 if the fields has milliseconds precision) 
+              # and subtract the current time in seconds since epoch.
+              expression: EXPIRES_TIMESTAMP / 1000 - STRFTIME('%s', 'now')
               language: sql
     ```
 
-----
-
-### SQL Server examples
-SQL Server supports the following date and time data types:
+- For values matching the subset of ISO 8601 supported by SQLite e.g. `2023-10-01T12:00:00`, `2023-10-01T12:00:00Z` or `2025-06-05T13:40:14.784000+02:00` you can use the `STRFTIME` function to convert the value to seconds since epoch and subtract the current time in seconds since epoch from it.
 
-- `date` - represented in Debezium as number of days since epoch (1970-01-01). Please note that due to the lack of time information, this method is not very accurate.
-  ```yaml
-    output:
-    - uses: redis.write
-      with:
-        data_type: hash
-        expire:
-          # Calculate the number of seconds equivalent to the number of days and subtract the current time in seconds since epoch.
-          expression: (event_date * 86400) - strftime('%s', 'now')
-          language: sql
-  ```
-
-- `datetime`, `smalldatetime` - represented in Debezium as number of milliseconds since epoch.
-  ```yaml
-    output:
-    - uses: redis.write
-      with:
-        data_type: hash
-        expire:
-          # Since event_datetime is in miliseconds, you must divide it by 1000 to convert it to seconds.
-          expression: event_datetime / 1000 - strftime('%s', 'now')
-          language: sql
-  ```
-- `datetime2` - similar to `datetime` but with higher precision. For `datetime2(0-3)` the representation is the same as for `datetime`. For `datetime2(4-6)` it is the number of microseconds since epoch. and for `datetime2(7)` it is the number of nanoseconds since epoch. You can use the same approach as for `datetime` but you need to divide by 1000, 1000000 or 1000000000 depending on the precision.
-
-- `time` - the time of milliseconds since midnight.
-  ```yaml
-    output:
-    - uses: redis.write
-      with:
-        data_type: hash
-        expire:
-          # Convert the time to seconds and subtract the current time in seconds since midnight.
-          expression: (event_time / 1000.0) -
-            (
-              CAST(strftime('%H', 'now') AS INTEGER) * 3600 +
-              CAST(strftime('%M', 'now') AS INTEGER) * 60 +
-              CAST(strftime('%S', 'now') AS INTEGER)
-            )
-          language: sql
-  ```
-- `datetimeoffset` - represented as a timestamp with timezone information, where the timezone is GMT
   ```yaml
   output:
     - uses: redis.write
       with:
         data_type: hash
         expire:
-          # Convert the time to seconds and subtract the current time in seconds since epoch.
-          expression: strftime('%s', event_datetimeoffset) - strftime('%s', 'now')
           language: sql
+          expression: STRFTIME('%s', EXPIRATION_TS) - STRFTIME('%s', 'now')
   ```
 
-<!-- TODO [ilianiliev-redis]: Test and document the dynamic expressions for the rest of the supported databases - MySQL, PostgresSQL, MongoDB -->
+For more examples on how to manipulate date and time values, see the [Formatting date and time values]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/formatting-date-and-time-values/">}}) page.
+