More work on types for tables examples. #779

duvenaud · 2022-03-04T23:32:07Z

This isn't finished yet, but could be merged now since it's self-contained.

Thanks to Adam's extensions, this is going smoothly so far. It's surprising how often the code is correct once it typechecks. Thanks to punning and lambdas, the code is about as terse as I think is possible. For example, one of the examples is written as:

> isTeenagerBuilder =
    function(r):
      12 < getValue(r, "age") and getValue(r, "age") < 20
    end
> buildColumn(students, "is-teenager", isTeenagerBuilder)

which translates to Dex as:

buildColumn students ##is_teenager \{age, ..._}.
  (12 < age ) && (age < 20)

I'm not sure whether to leave the original examples in as comments. It's nice to see how much shorter the Dex versions are, but the examples and example outputs are pretty long.

I played a bit with indexing by sets of strings, but it's a bit more awkward than I expected. Things like adding a row, or concatenating two tables then sometimes mean changing the index type, which requires lots of type-level computation which isn't supported. I tried to make a set-erased SetIxSet, kind of like an unordered list, but then I don't think there's enough information available at compile time for it to fulfill the Ix interface.

I've been getting random segfaults that crash the web notebook, but everything always works when I restart.

I feel like the records system is getting a bit heavy on the new syntax. Could we potentially replace Fields with simply List Label or n=>Label, and avoid introducing the syntax for creating new Fields?

There are also a couple of minor bugs or inconsistencies here and there in the original examples.

apaszke

Nice!

apaszke · 2022-03-08T14:27:23Z

examples/data-frames.dx

+  ]
+
+
+'# Experiment: Indexing by sets


Can we please move the experiment to the end?

Will do. I was originally planning to do the entire file twice, once with indexing by row number and once indexing by row elements, but indexing by row elements doesn't work yet.

apaszke · 2022-03-08T14:27:43Z

examples/data-frames.dx

-- | "Bob"   | 12  | 8     | 9     | 77      | 7     | 9     | 87    |
-- | "Alice" | 17  | 6     | 8     | 88      | 8     | 7     | 85    |
-- | "Eve"   | 13  | 7     | 9     | 84      | 8     | 8     | 77    |
+:p addRows students [{name="Colton", age=19, favColor="blue"}]


nit: I don't think you have to use :p anymore.

Also, once this is done, can you please check in the outputs, so that we can turn it into a test?

Heh, I was only using it because I thought there was a reason you guys were using it.

No, I think we should really delete it... I don't see a good reason to keep it.

apaszke · 2022-03-08T14:31:53Z

examples/data-frames.dx

-- | true     | false | false |
-- | true     | false | true  |
+-- | =True     | =False | =False |
+-- | =True     | =False | =True  |


Weird formatting? Why the = before booleans?

Whoops, that's left over from a find-and-replace when I was formatting all the tables to dex tables.

apaszke · 2022-03-08T14:32:24Z

examples/data-frames.dx

+-- | =True     | =False | =True  |
+
+
+-- :p crossJoin students petiteJelly


Cross join examples are now attached to subTable, not to crossJoin

Ah, got it.

apaszke · 2022-03-08T14:37:18Z

examples/data-frames.dx

+  (AsList num_found found_table) = argFilter (\x. x == v) xs
+  case num_found == 0 of
+    True -> Nothing
+    False -> Just found_table.(unsafeFromOrdinal _ 0)


You could also do this:

data First a = MkFirst (Maybe a) def FirstMonoid (a:Type) : Monoid (First a) = named-instance result : Monoid (First a) mempty = MkFirst Nothing mappend = \x y. case x of Just xv -> x Nothing -> y result def findInUnordered {a n} [Eq a] (xs:n=>a) (v:a) : Maybe n = MkFirst ans = withAccum \ref. for_ i. case xs.i == v of True -> ref += MkFirst (Just n) False -> () ans

Nice, yes I guess that'd be a bit more efficient. I wonder if someday we can have some sort of Lattice typeclass and corresponding effect which lets the compiler know about short-circuiting completions.

That would be pretty neat! We could have a flavor of Accum that works over a lattice and short-circuits (when the computation is otherwise pure?)!

Yes, I think that'd be a good showcase of Dex's effect-aware compilation. I've come across a couple of other examples that could in principle be sped up using a parallel short-circuit:

The any and all functions.

The Ord instance for lists which returns at the location of first difference

apaszke · 2022-03-08T14:40:42Z

examples/data-frames.dx

@@ -347,43 +345,35 @@ def getValue {f a} (c:Label) (r:{@c:a & ...f}) : a =
 -- ### (overloading 1/2) `getColumn :: t:Table * n:Number -> vs:Seq<Value>`

 -- ### (overloading 2/2) `getColumn :: t:Table * c:ColName -> vs:Seq<Value>`
-def getColumn {f n a} (c:Label) (t:n=>{@c:a & ...f}) : n=>a = todo
+
+-- Define above.


nit: Defined

apaszke · 2022-03-08T14:46:16Z

examples/data-frames.dx

-- > head(students, -2)
+
+:p selectColumns {favColor:_ ? age:_} students
+:p selectColumns {final:_ ? name:_ ? midterm:_} gradebook


Doesn't this replace subTable from above? I think at some point the benchmark authors might have renamed subTable to selectColumns and only forgot to update that one example. Let's use this name instead.

apaszke · 2022-03-08T14:47:30Z

examples/data-frames.dx

+  (UnsafeAsSet _ uniquetable) = toSet t
+  AsList _ uniquetable
+
+:p distinct students


Does that work? I would imagine that we don't have an Ord instance for records?

It doesn't work yet. I think 3 of the examples parse but still fail with one error or another that I'm hoping to chat with you about.

Sorry, I should have been clearer that this PR was in the same spirit as the one you merged which introduced this file, which is "this doesn't fully work yet but is better than what came before." I learned it from you! :)

Oh, WIP is totally fine. I was only a bit confused, because it looks as if it was supposed to work, but I couldn't figure out how :)

apaszke · 2022-03-08T14:52:11Z

examples/data-frames.dx

+  xlists = for i:n. (AsList 1 ([(xs.i, 1)]))
+  reduce (AsList 0 []) mergeUniqueSortedListsWithCounts xlists
+
+def count {n a f} [Ord a] (c:Label) (t:n=>{@c:a & ...f}) : List {value:a & count:Int} =


This is another place where a mutable HashMap would probably be a better fit

apaszke · 2022-03-08T14:53:17Z

examples/data-frames.dx

+          True ->
+            concat [allExceptLast xs,
+                    (AsList 1 [(last_x, concat [x_b_list, y_b_list])]),
+                    unsafeAllExceptFirst ys]


This function is duplicated way too many times. Is there a nice generic version we could write once?

Yes, certainly. I'll unify these the next time I work on this. I just wanted to write the specific instances first so that I'd know what the general version needed to look like.

duvenaud added 4 commits March 4, 2022 18:29

More work on types for tables examples.

4918a17

Added leftJoin and count to dataframes example.

ac79542

Added argsort with tests.

e5b564f

Added groupBy and fixed argsort tests.

46ed721

duvenaud changed the title ~~[WIP] More work on types for tables examples.~~ More work on types for tables examples. Mar 7, 2022

apaszke reviewed Mar 8, 2022

View reviewed changes

apaszke force-pushed the main branch from 46b8727 to 8db43fc Compare May 13, 2022 14:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More work on types for tables examples. #779

More work on types for tables examples. #779

duvenaud commented Mar 4, 2022 •

edited

Loading

apaszke left a comment

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 9, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 9, 2022

duvenaud Mar 9, 2022

apaszke Mar 8, 2022

apaszke Mar 8, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022

apaszke Mar 9, 2022

apaszke Mar 8, 2022

apaszke Mar 8, 2022

duvenaud Mar 8, 2022 •

edited

Loading

		-- \| =True \| =False \| =True \|


		-- :p crossJoin students petiteJelly

More work on types for tables examples. #779

Are you sure you want to change the base?

More work on types for tables examples. #779

Conversation

duvenaud commented Mar 4, 2022 • edited Loading

apaszke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

duvenaud Mar 8, 2022 • edited Loading

Choose a reason for hiding this comment

duvenaud commented Mar 4, 2022 •

edited

Loading

duvenaud Mar 8, 2022 •

edited

Loading