Implement limit push down for IcebergTableProvider

[krinart]

Which issue does this PR close?

N/A

What changes are included in this PR?

Previously _limit was ignored in IcebergTableProvider::scan:

iceberg-rust/crates/integrations/datafusion/src/table/mod.rs

Lines 149 to 163 in aad9e2e

	_state: &dyn Session,
	projection: Option<&Vec<usize>>,
	filters: &[Expr],
	_limit: Option<usize>,
	) -> DFResult<Arc<dyn ExecutionPlan>> {
	Ok(Arc::new(IcebergTableScan::new(
	self.table.clone(),
	self.snapshot_id,
	self.schema.clone(),
	projection,
	filters,
	)))
	}
	fn supports_filters_pushdown(

This PR propagates limit all the way to the ArrowReaderBuilder.

Note: limit push down is only applied to each batch which means that IcebergTableProvider::scan may potentially return more records than specified by limit.

Which is OK according to TableProvider::scan documentation:

If limit is specified, must only produce at least this many rows, (though it may return more).

Are these changes tested?

Unit tests

[ZENOTME]

Should we enable with_page_index as suggested by doc: https://docs.rs/parquet/latest/parquet/arrow/arrow_reader/struct.ArrowReaderBuilder.html#method.with_limit

[krinart]

Thanks! I extended should_load_page_index logic and ArrowReaderOptions is initialized with with_page_index(should_load_page_index).

[ZENOTME]

Thanks @krinart

[krinart]

Happy to help @ZENOTME and thanks for the feedback!
Any suggestions on how to proceed and merge this PR?

[krinart]

Hi. Anything else I can do on my side to get this merged?

[Xuanwo]

Thank you for working on this!

[Xuanwo]

hi, @krinart please allow me to edit this PR so we can resolve the conflicts and merge it.

[krinart]

Hey @Xuanwo, apologies for a late reply.

I just resolved all the conflicts. Let me know if there's anything else I can do. Thanks!

[lukekim]

@Xuanwo anything left to get this merged?

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.