Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to convert JSON to Avro #26

Open
havnar opened this issue Aug 1, 2018 · 2 comments
Open

Failed to convert JSON to Avro #26

havnar opened this issue Aug 1, 2018 · 2 comments

Comments

@havnar
Copy link

havnar commented Aug 1, 2018

I'm getting the following exception:
Caused by: org.apache.avro.AvroRuntimeException: Field deviceid type:LONG pos:0 not set and has no default value
for Json entry "device_id":6854

the schema for this field looks like this:
{"name":"deviceid","type":"long","doc":" The id of the device.","aliases":["device_id","device/id"]}

Is this because the codebase does not take aliases into account?

@adamdubiel
Copy link
Collaborator

We never tried aliases. However they should work out of box if Avro library supports them.

@frankgrimes97
Copy link

We've hit this same issue. Would you be open to a PR to address it? I've tested the following patch locally and it appears to work.

$ git diff
diff --git a/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java b/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java
index d1c4547..76ae90f 100644
--- a/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java
+++ b/converter/src/main/java/tech/allegro/schema/json2avro/converter/JsonGenericRecordReader.java
@@ -62,11 +62,21 @@ public class JsonGenericRecordReader {
     }
 
     private GenericData.Record readRecord(Map<String, Object> json, Schema schema, Deque<String> path) {
-        GenericRecordBuilder record = new GenericRecordBuilder(schema);
+        final GenericRecordBuilder record = new GenericRecordBuilder(schema);
+        final Map<String,Field> fieldAliasMap = new HashMap<>(1);
+        for (Field field: schema.getFields()) {
+            for (String alias: field.aliases()) {
+                fieldAliasMap.put(alias, field);
+            }
+        }
         json.entrySet().forEach(entry -> {
-            Field field = schema.getField(entry.getKey());
+            final String entryKey = entry.getKey();
+            final Field field = schema.getField(entryKey);
             if (field != null) {
                 record.set(field, read(field, field.schema(), entry.getValue(), path, false));
+            } else if (fieldAliasMap.containsKey(entryKey)) {
+                final Field aliasedField = fieldAliasMap.get(entryKey);
+                record.set(aliasedField, read(aliasedField, aliasedField.schema(), entry.getValue(), path, false));
             } else if (unknownFieldListener != null) {
                 unknownFieldListener.onUnknownField(entry.getKey(), entry.getValue(), PathsPrinter.print(path, entry.getKey()));
             }

frankgrimes97 added a commit to frankgrimes97/json-avro-converter that referenced this issue Jun 19, 2020
frankgrimes97 added a commit to frankgrimes97/json-avro-converter that referenced this issue Jun 19, 2020
frankgrimes97 added a commit to frankgrimes97/json-avro-converter that referenced this issue Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants