Deprecate custom_parse in favor of custom_func

README.md

@@ -38,7 +38,7 @@ Examples:
 - [XML Examples](extensions/omniv21/samples/xml).
 - [EDI Examples](extensions/omniv21/samples/edi).
 - [Custom File Format](extensions/omniv21/samples/customfileformats/jsonlog)
-- [Custom Parse](extensions/omniv21/samples/customparse)
+- [Custom Funcs](extensions/omniv21/samples/customfuncs)
 
 In the example folders above you will find pairs of input files and their schema files. Then in the
 `.snapshots` sub directory, you'll find their corresponding output files.
@@ -64,6 +64,8 @@ situations.
 - Golang 1.14
 
 ## Recent Major Feature Additions/Changes
+- Deprecated `custom_parse` in favor of `custom_func` (`custom_parse` is still usable for
+back-compatibility, it is just removed from all public docs and samples).
 - Added `NonValidatingReader` EDI segment reader.
 - Added fixed-length file format support in omniv21 handler.
 - Added EDI file format support in omniv21 handler.

doc/programmability.md

@@ -1,7 +1,6 @@
 * [Programmability of Omniparser](#programmability-of-omniparser)
   * [Out\-of\-Box Basic Use Case](#out-of-box-basic-use-case)
   * [Add A New custom\_func](#add-a-new-custom_func)
-  * [Add A New custom\_parse](#add-a-new-custom_parse)
   * [Add A New File Format](#add-a-new-file-format)
   * [Add A New Schema Handler](#add-a-new-schema-handler)
   * [Put All Together](#put-all-together)
@@ -77,31 +76,6 @@ Each `custom_func` must be a Golang function with the first param being `*transf
 params can be of any type, as long as they will match the types of data that are fed into the function
 in `transform_declarations`.
 
-## Add A New `custom_parse`
-
-There are several ways to customize transform logic, one of which is using the all mighty `custom_func`
-`javascript` (or its silibing `javascript_with_context`), see details
-[here](./use_of_custom_funcs.md#javascript-and-javascript_with_context).
-
-However currently we don't support multi-line javascript (yet), which makes writing complex transform
-logic in a single line javascript difficult to read and debug. Also there are situations where schema
-writers want the following:
-- native Golang code transform logic
-- logging/stats
-- better/thorough test coverage
-- more complexed operations like RPCs calls, encryption, etc, which isn't really suited/possible for
-javascript to handle.
-
-`custom_parse` provides an in-code transform plugin mechanism. In addition to a number of built-in
-transforms, such as field, `const`, `external`, `object`, `template`, `array`, and `custom_func`,
-`custom_parse` allows schema writer to be able to provide a Golang function that takes in the
-`*idr.Node` at the current IDR cursor (see more about IDR cursoring
-[here](./xpath.md#data-context-and-anchoring)), does whatever processing and transforms as it sees
-fit, and returns whatever the desired result to be embedded in place of the `custom_parse`.
-
-[This sample](../extensions/omniv21/samples/customparse/sample_test.go) gives a very detailed demo
-of how `custom_parse` works.
-
 ## Add A New File Format
 
 While built-in `omni.2.1` schema handler already supports most popular file formats in a typical

doc/transforms.md

@@ -134,16 +134,13 @@ examples:
 - Custom Function Call (**custom_func**): e.g. `{ "custom_func": {...} }`. See more details about
 `custom_func` transform directive [here](./use_of_custom_funcs.md).
 
-- Custom Parse (**custom_parse**): e.g. `{ "custom_parse": "<custom parse id>" }`. See more details about
-`custom_parse` transform directive [here](./programmability.md).
-
 ## Miscellaneous
 
 Several attributes can be specified on some or all transform directives:
 
 1. `xpath` (or `xpath_dynamic`) can be used for data extraction or IDR cursor anchoring with the following
 transform types: field (in fact field has nothing else but an `xpath` or `xpath_dynamic`), `object`,
-`template`, `custom_func` and `custom_parse`. See more details about use of `xpath` (or `xpath_dynamic`)
+`template`, and `custom_func`. See more details about use of `xpath` (or `xpath_dynamic`)
 [here](./xpath.md).
 
 2. `type` tells omniparser the result from the transform needs a type cast. Supported type cast types are:

doc/xpath.md

@@ -81,10 +81,10 @@ The most common use of `xpath` is for data extraction. Consider again the sample
 ```
 
 The `xpath` attributes on `"order_id"`, `"customer_id"`, and `"country"` tell omniparser where to get
-the field string data from. When `xpath` **not** appearing with `object`, `template`, `custom_func`, or
-`custom_parse`, then it is a data extraction directive telling omniparser to extract the text data at the
-location specified by the `xpath` query. Note in this situation, omniparser will require the result set of
-such `xpath` queries to be of a single node: if such `xpath` query results in more than one node, omniparser
+the field string data from. When `xpath` **not** appearing with `object`, `template`, or `custom_func`,
+then it is a data extraction directive telling omniparser to extract the text data at the location
+specified by the `xpath` query. Note in this situation, omniparser will require the result set of such
+`xpath` queries to be of a single node: if such `xpath` query results in more than one node, omniparser
 will fail the current record transform (but will continue onto the next one as this isn't considered fatal).
 
 ## Data Context and Anchoring
@@ -191,8 +191,7 @@ Now let's go through the schema and input together to see how `xpath` anchoring
     for a `custom_func`, then everything inside the `custom_func`, namely those argument transforms, are all
     anchored on the cursor position prescribed by the `xpath`.
 
-When `xpath` is used for anchoring and cursoring, it can appear with `object`, `template`, `custom_func`, and
-`custom_parse`.
+When `xpath` is used for anchoring and cursoring, it can appear with `object`, `template`, and `custom_func`.
 
 ## Static and Dynamic XPath Queries
 
@@ -280,8 +279,8 @@ for `color` field in the output, we need to dynamically construct an XPath query
 needed for `size` field data extraction.
 
 `xpath_dynamic` is used in such a situation. It basically says, unlike `xpath` is always a constant and static
-string value, `xpath_dynamic` is computed, by either `custom_func`, or `custom_parse`, or `template`, or
-`external`, or `const`, or another `xpath` direct data extraction.
+string value, `xpath_dynamic` is computed, by either `custom_func`, or `template`, or `external`, or
+`const`, or another `xpath` direct data extraction.
 
 `xpath_dynamic` can be used everywhere `xpath` is used, except on `FINAL_OUTPUT`. `FINAL_OUTPUT` can only
 use `xpath`.
@@ -304,7 +303,7 @@ Depending on which transform is in play, different outcomes, including error, ca
     - Example: `"FINAL_OUTPUT": { "xpath": "/publishers/*", "object": {`
     - The result set can be either empty, or of one node, or of multiple nodes.
 
-- `xpath`/`xpath_dynamic` used in `object`, `custom_func`, `custom_parse`, `template` transform
+- `xpath`/`xpath_dynamic` used in `object`, `custom_func`, `template` transform
 (other than `FINAL_OUTPUT` or directly under an `array` transform):
 
     - Example: `"contact": { "xpath": "PATH/TO/CONTACT", "object": {`

extensions/omniv21/ingester.go

@@ -14,7 +14,7 @@ import (
 type ingester struct {
 	finalOutputDecl  *transform.Decl
 	customFuncs      customfuncs.CustomFuncs
-	customParseFuncs transform.CustomParseFuncs
+	customParseFuncs transform.CustomParseFuncs // Deprecated.
 	ctx              *transformctx.Ctx
 	reader           fileformat.FormatReader
 }

extensions/omniv21/samples/customfuncs/sample_schema.json

@@ -0,0 +1,14 @@
+{
+    "parser_settings": {
+        "version": "omni.2.1",
+        "file_format_type": "xml"
+    },
+    "transform_declarations": {
+        "FINAL_OUTPUT": { "xpath": "/Employees/Employee", "object": {
+            "employee_id": { "xpath": "ID" },
+            "personal_details": { "xpath": "ID", "custom_func": { "name": "employee_personal_details_lookup" }},
+            "business_details": { "xpath": "ID", "custom_func": { "name": "employee_business_details_lookup" }},
+            "team": { "xpath": "ID", "custom_func": { "name": "employee_team_lookup" }}
+        }}
+    }
+}

extensions/omniv21/samples/customparse/sample_test.go → extensions/omniv21/samples/customfuncs/sample_test.go

@@ -1,4 +1,4 @@
-package customparse
+package customfuncs
 
 import (
 	"io"
@@ -13,8 +13,9 @@ import (
 	"github.com/stretchr/testify/assert"
 
 	"github.com/jf-tech/omniparser"
+	"github.com/jf-tech/omniparser/customfuncs"
 	"github.com/jf-tech/omniparser/extensions/omniv21"
-	"github.com/jf-tech/omniparser/extensions/omniv21/transform"
+	v21 "github.com/jf-tech/omniparser/extensions/omniv21/customfuncs"
 	"github.com/jf-tech/omniparser/idr"
 	"github.com/jf-tech/omniparser/transformctx"
 )
@@ -37,13 +38,14 @@ func TestSample(t *testing.T) {
 		schemaFileReader,
 		omniparser.Extension{
 			CreateSchemaHandler: omniv21.CreateSchemaHandler,
-			CreateSchemaHandlerParams: &omniv21.CreateParams{
-				CustomParseFuncs: transform.CustomParseFuncs{
+			CustomFuncs: customfuncs.Merge(
+				customfuncs.CommonCustomFuncs,
+				v21.OmniV21CustomFuncs,
+				customfuncs.CustomFuncs{
 					"employee_personal_details_lookup": employeePersonalDetailsLookup,
 					"employee_business_details_lookup": employeeBusinessDetailsLookup,
 					"employee_team_lookup":             employeeTempLookup,
-				},
-			},
+				}),
 		})
 	assert.NoError(t, err)
 	transform, err := schema.NewTransform(inputFileBaseName, inputFileReader, &transformctx.Ctx{})
@@ -71,7 +73,7 @@ func TestSample(t *testing.T) {
 func employeePersonalDetailsLookup(_ *transformctx.Ctx, node *idr.Node) (interface{}, error) {
 	id := node.InnerText()
 	// Pretend some complex logic and/or RPC calls...
-	// This custom_parse demonstrates how to return a complex object with map[string]interface{}
+	// This custom_func demonstrates how to return a complex object with map[string]interface{}
 	idInt, err := strconv.Atoi(id)
 	if err != nil {
 		return nil, err
@@ -91,7 +93,7 @@ func employeePersonalDetailsLookup(_ *transformctx.Ctx, node *idr.Node) (interfa
 func employeeBusinessDetailsLookup(_ *transformctx.Ctx, node *idr.Node) (interface{}, error) {
 	id := node.InnerText()
 	// Pretend some complex logic and/or RPC calls...
-	// This custom_parse demonstrates how to return a complex object with golang struct
+	// This custom_func demonstrates how to return a complex object with golang struct
 	type employeeReview struct {
 		Year   int    `json:"year"`
 		Rating string `json:"rating"`

extensions/omniv21/schemahandler.go

@@ -25,7 +25,8 @@ const (
 // CreateParams allows user of this 'omni.2.1' schema handler to provide creation customization.
 type CreateParams struct {
 	CustomFileFormats []fileformat.FileFormat
-	CustomParseFuncs  transform.CustomParseFuncs
+	// Deprecated.
+	CustomParseFuncs transform.CustomParseFuncs
 }
 
 // CreateSchemaHandler parses, validates and creates an omni-schema based handler.

extensions/omniv21/transform/decl.go

@@ -18,7 +18,7 @@ const (
 	kindObject      kind = "object"
 	kindArray       kind = "array"
 	kindCustomFunc  kind = "custom_func"
-	kindCustomParse kind = "custom_parse"
+	kindCustomParse kind = "custom_parse" // Deprecated
 	kindTemplate    kind = "template"
 )
 
@@ -83,7 +83,7 @@ type Decl struct {
 	XPathDynamic *Decl `json:"xpath_dynamic,omitempty"`
 	// CustomFunc specifies the input element is a custom function.
 	CustomFunc *CustomFuncDecl `json:"custom_func,omitempty"`
-	// CustomParse specifies the input element is to be custom parsed.
+	// CustomParse specifies the input element is to be custom parsed. Deprecated.
 	CustomParse *string `json:"custom_parse,omitempty"`
 	// Template specifies the input element is a template.
 	Template *string `json:"template,omitempty"`

extensions/omniv21/transform/invokeCustomParse.go

@@ -7,10 +7,11 @@ import (
 	"github.com/jf-tech/omniparser/transformctx"
 )
 
-// CustomParseFuncType represents a custom_parse function type.
+// CustomParseFuncType represents a custom_parse function type. Deprecated. Use customfuncs.CustomFuncType.
 type CustomParseFuncType func(*transformctx.Ctx, *idr.Node) (interface{}, error)
 
-// CustomParseFuncs is a map from custom_parse names to an actual custom parse functions.
+// CustomParseFuncs is a map from custom_parse names to an actual custom parse functions. Deprecated. Use
+// customfuncs.CustomFuncs.
 type CustomParseFuncs = map[string]CustomParseFuncType
 
 func (p *parseCtx) invokeCustomParse(customParseFn CustomParseFuncType, n *idr.Node) (interface{}, error) {

extensions/omniv21/transform/parse.go

@@ -15,8 +15,8 @@ import (
 type parseCtx struct {
 	transformCtx          *transformctx.Ctx
 	customFuncs           customfuncs.CustomFuncs
-	customParseFuncs      CustomParseFuncs
-	disableTransformCache bool // by default we have caching on. only in some tests we turn caching off.
+	customParseFuncs      CustomParseFuncs // Deprecated.
+	disableTransformCache bool             // by default we have caching on. only in some tests we turn caching off.
 	transformCache        map[string]interface{}
 }
 

extensions/omniv21/transform/validate.go

@@ -16,7 +16,7 @@ import (
 type validateCtx struct {
 	Decls            map[string]*Decl `json:"transform_declarations"`
 	customFuncs      customfuncs.CustomFuncs
-	customParseFuncs CustomParseFuncs
+	customParseFuncs CustomParseFuncs // Deprecated.
 	declHashes       map[string]string
 }