Describe null handling in anti join

[obacht17]

Current state

The documentation says an ANTI JOIN and WHERE NOT IN (...) are equivalent. This is not true when the right table of the ANTI JOIN / subquery of the NOT IN (...) does contain NULL values.

Detailed demonstration

create table t1 (id1 integer);
create table t2 (id1 integer);
insert into t1 select id1 from generate_series(1, 1000) tmp1(id1);
insert into t2 select id1 from generate_series(1, 100) tmp1(id1);
insert into t2 values(null);  -- a null value on the right side is the key here

select count(*) as q1 from t1 where id1 not in (select id1 from t2);  
--> q1: 0

select count(*) as q2 from t1 anti join t2 using (id1);
--> q2: 900
--> i.e. WHERE id1 NOT IN (...) is not equivalent to ANTI JOIN

select count(*) as q3 from t1 where id1 not in (select id1 from t2 where id1 is not null);
--> q3: 900
--> i.e. we can fix this by excluding NULLs from the subquery which do not match in the anti join.

Verification using Postgres

To make sure I did not step on a bug in DuckDB, I tried q1 and q3 in Postgres (online tool, PostgreSQL 14.11). The results were the same as in DuckDB 1.1.1. Please note that Postgres parses ANTI JOIN syntax in q2, but does definitely not apply an ANTI JOIN, but an INNER JOIN.

[szarnyasg]

Hi @obacht17, thanks for reporting this an submitting a PR with the clarification. This is indeed an important thing to mention.

Please note that Postgres parses ANTI JOIN syntax in q2, but does definitely not apply an ANTI JOIN, but an INNER JOIN.

Postgres interprets the string anti as an alias for t1, i.e. these two are equivalent:

select count(*) as q2 from t1    anti join t2 using (id1);
select count(*) as q2 from t1 AS anti join t2 using (id1);