profile
viewpoint
Joshua C. Randall jrandall Genomics plc Cambridge, UK

jrandall/cambridge-challenge.appspot.com 3

Google appengine application for hosting clues & hints for a puzzle/treasure hunt (primarily accessed from mobile location-aware devices)

jcbarret/puzzleboss 1

Some code for the Putz Tetazoo team working on the MIT Mystery Hunt

jrandall/freeciv-web 1

Freeciv.net is an Open Source strategy game which can be played online against other players, or in single player mode against AI opponents.

jrandall/ggplot2 1

An implementation of the Grammar of Graphics in R

jrandall/ansible 0

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications— automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

jrandall/arvados 0

an open source platform for managing and analyzing biomedical big data

jrandall/baton 0

iRODS client programs and API

jrandall/bhaps 0

converters from text-based haplotype matrix to binary haplotype matrix, and for sampling haplotypes into genotypes

jrandall/boto 0

For the latest version of boto, see https://github.com/boto/boto3 -- Python interface to Amazon Web Services

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha b252f5e88e8452ba567930899d09df0fff5e1910

cargo fmt

view details

Joshua C. Randall

commit sha 55c0c3cfcc350a06b69256c4b2ca41d30a03824f

add safety docs to unsafe functions in example-extension and example-embedded-extension

view details

Joshua C. Randall

commit sha 069fc87fa12d5ddd220d1550618f0767ec403b39

cargo fmt

view details

Joshua C. Randall

commit sha b4ae3e0abaf96ddd792837c53d970d91c123c4cc

fix doctests in src/loadable_extension.rs

view details

Joshua C. Randall

commit sha 3819e07a55d39430685fed97bbf716335403b835

access sqlite3_api_routines fields in generated wrapper functions using addr_of! and read to avoid undefined behaviour when runtime sqlite3_api_routines is shorter than expected at compile time

view details

Joshua C. Randall

commit sha 7822a0b54ed6b7f8f66ce13aa0622640707f1061

Merge branch 'buildtime-bindgen-wasm32-v0.26.2' into integration/lentil-v0.26.2

view details

Joshua C. Randall

commit sha 66052e282eef339b895ae9852b6c12b1c89aa239

Merge branch 'loadable-extensions-v0.26.2' into integration/lentil-v0.26.2

view details

Joshua C. Randall

commit sha e7197ae0e781cc0e0439b4c46e0be4431e37ee7f

Merge branch 'pub-raw-statement-v0.26.2' into integration/lentil-v0.26.2

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha b252f5e88e8452ba567930899d09df0fff5e1910

cargo fmt

view details

Joshua C. Randall

commit sha 55c0c3cfcc350a06b69256c4b2ca41d30a03824f

add safety docs to unsafe functions in example-extension and example-embedded-extension

view details

Joshua C. Randall

commit sha 069fc87fa12d5ddd220d1550618f0767ec403b39

cargo fmt

view details

Joshua C. Randall

commit sha b4ae3e0abaf96ddd792837c53d970d91c123c4cc

fix doctests in src/loadable_extension.rs

view details

Joshua C. Randall

commit sha 3819e07a55d39430685fed97bbf716335403b835

access sqlite3_api_routines fields in generated wrapper functions using addr_of! and read to avoid undefined behaviour when runtime sqlite3_api_routines is shorter than expected at compile time

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 3819e07a55d39430685fed97bbf716335403b835

access sqlite3_api_routines fields in generated wrapper functions using addr_of! and read to avoid undefined behaviour when runtime sqlite3_api_routines is shorter than expected at compile time

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha b4ae3e0abaf96ddd792837c53d970d91c123c4cc

fix doctests in src/loadable_extension.rs

view details

push time in a month

pull request commentrusqlite/rusqlite

Add support for sqlite loadable extensions

A better way is probably to store and read into the functions themselves, rather the the API. This would also give us a path forward to eventually supporting dlopen/dlsym.

Specifcally, rather than a single AtomicPtr<sqlite3_api_routines>, we'd effectively have one Atomic<Option<unsafe extern "C" fn(...)>>3 per field. Then, when initializing the extension API, we'd call libversion_number from api pointer, and use that to figure out which fields should be valid, and populate the supported function pointers.

I like this approach more, and was kind of planning on doing it anyway for dlopen/dlsym support. It should be easier after I overhaul libsqlite3-sys, though.

I think I see what you are suggesting here, and the new loadable_extension_init and loadable_extension_embedded_init functions provide the extension point where this could be achieved.

Your approach would have the benefit of doing this version check once at initialization rather than (potentially) on every call to every API routine. I had thought the current approach could be extended by memoizing (really just Once-ing) the runtime checks for each function, which could also address those concerns (probably with about the same overhead as doing the runtime check against the Option).

I had also thought about extending the bindings to provide something like an *_is_supported function for each of the routines, so that consumers of the library can anticipate and avoid the panic that would otherwise be caused by the runtime checks (i.e. by having some fallback behaviour when certain routines are not available). However, it was starting to feel like there is too much logic being written into the bindings here, and I'd like to be able to push a

Your proposed interface would address that cleanly if we provide the struct representing the API routines to consumers so that they can check against the Option for the routine of interest prior to invoking it.

I guess what I'm not clear on here is how we would provide this to consumers of libsqlite3-sys such as rusqlite -- would you propose that the main interface that libsqlite3-sys provides to its consumers (including rusqlite) be refactored so that all accesses to sqlite3_* methods go through an API methods struct such as the one you suggest? I guess that would have the possible benefit of decoupling the sqlite3 API routines that the libsqlite3-sys rust library is aware exist from those that are acutally present in the version of sqlite3 that its bindings are built against. Even when not building an extension, an entrypoint could populate the API routines structure when statically compiled, such that those that are present in the sqlite it is built against are Some(fn ... ) and newer ones that aren't supported are None. This would push the responsibility for adapting to different runtime versions of sqlite to the consumers of libsqlite3-sys, but it would also obviously give them the responsibility for handling situations in which certain routines are not available appropriately.

Or, alternatively, would we continue to offer (unwrapped) sqlite3_* methods to consumers of libsqlite3-sys but back them using the proposed methods structure in certain situations, with a default way of handling routines (that would probably be to handle a None invocation via a panic?) but also offer the wrapped API routines Options interface as an alternative way for consumers to access API routines when they are able to provide fallback handling in the case of API routines that are None?

jrandall

comment created time in a month

pull request commentrusqlite/rusqlite

Add support for sqlite loadable extensions

I am genuinely concerned about the safety issue here with versions... I don't think requiring the version match the generated bindings is okay, since (for example), I'm planning on getting rid of as many of the bindings files that aren't prebundled as I can (buildtime-bindgen will still exist) when I have time1. This fits with what the other libsqlite3-sys using crates have been asking for a while, and also will simplify things and make fewer changes breaking for people who separately compile sqlite.

Anyway, a way forward here. Hm. I have some code where I tried to parse the C sqlite3ext.h to get the versions, which worked fine (the original motivation for it was unrelated to extensions, actually...)

@thomcc I am not sure if you saw yet that I have implemented this in the current branch. The build script for libsqlite3-sys includes code to track the version comments in the sqlite headers and set a version check Option for each API function.

Parse rust-bindgen bindings (generated with comments from sqlite3ext.h): https://github.com/rusqlite/rusqlite/blob/069fc87fa12d5ddd220d1550618f0767ec403b39/libsqlite3-sys/build.rs#L968-L1013

Generate runtime version checks for all API routines after the base set that are assumed to work in any version: https://github.com/rusqlite/rusqlite/blob/069fc87fa12d5ddd220d1550618f0767ec403b39/libsqlite3-sys/build.rs#L1279-L1332

jrandall

comment created time in a month

pull request commentrusqlite/rusqlite

Add support for sqlite loadable extensions

P.S. So, I hate to be that guy and ask, but is this the PR that had all the licensing headaches? My feeling there hasn't really changed.

Yes, it was. My employer owns the original code, and we had originally contributed it with a copyright statement for the company in the source files in which we made contributions (this is the company's usual policy and was also consistent with the state of copyright in the project at the time). Since opening the first PR on this (#532), the project changed its policy and rewrote all of the copyright statements to reference "The rusqlite developers". That unfortunately (and unintentionally as far as I can tell) resulted in all of my employer's copyright statements being removed when the default branch was merged into one of the subsequent PRs.

I subsequently went to my employer's legal team and got permission to contribute code with copyright statements attributed to "The rusqlite developers" and after some internal debate I was able to get them to agree to that with the provision that our contributions be attributed in an appropriate place such as in documentation or a CONTRIBUTORS file.

The current PR includes a (new) CONTRIBUTORS.md that would satisfy that requirement (https://github.com/rusqlite/rusqlite/blob/069fc87fa12d5ddd220d1550618f0767ec403b39/CONTRIBUTORS.md). However, we are willing to be as flexible as we can be in terms of fitting in with how the project would like to recognize non-individual contributors. If you do not agree that the proposed CONTRIBUTORS.md is the right solution, then please suggest an alternative.

It is my understanding that the intent is that the use of "The rusqlite developers" is intended to recognize (generally) all individuals who have contributed to the project, and that many such contributors can be identified through the git commit history, but in this case that would recognize me rather than my employer, and it is my employer who owns the work.

Will the current proposed CONTRIBUTORS.md file work for the project? If so, it might be nice to add a few more organizational contributors to it, both to recognise the support they have given the project and also so that it doesn't look like my employer is a more major contributor than they are.

If this won't work, do you have an alternative to suggest?

jrandall

comment created time in a month

Pull request review commentrusqlite/rusqlite

Add support for sqlite loadable extensions

 mod bindings {         let target_arch = std::env::var("TARGET").unwrap();         let host_arch = std::env::var("HOST").unwrap();         let is_cross_compiling = target_arch != host_arch;-+        let blocklist_va_list_functions = &vec![+            "sqlite3_vmprintf",+            "sqlite3_vsnprintf",+            "sqlite3_xvsnprintf",+            "sqlite3_str_vappendf",+        ];         // Note that when generating the bundled file, we're essentially always         // cross compiling.         if generating_bundled_bindings() || is_cross_compiling {-            // Get rid of va_list, as it's not+            // get rid of blocklisted functions that use va_list+            for fn_name in blocklist_va_list_functions {+                bindings = bindings.blocklist_function(fn_name);+                // Remove once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+                #[cfg(feature = "loadable_extension")]+                {+                    bindings_with_comments = bindings_with_comments.blocklist_function(fn_name);+                }+            }+            // Get rid of va_list             bindings = bindings-                .blocklist_function("sqlite3_vmprintf")-                .blocklist_function("sqlite3_vsnprintf")-                .blocklist_function("sqlite3_str_vappendf")                 .blocklist_type("va_list")                 .blocklist_type("__builtin_va_list")                 .blocklist_type("__gnuc_va_list")-                .blocklist_type("__va_list_tag")                 .blocklist_item("__GNUC_VA_LIST");+            // Remove once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+            #[cfg(feature = "loadable_extension")]+            {+                bindings_with_comments = bindings_with_comments+                    .blocklist_type("va_list")+                    .blocklist_type("__builtin_va_list")+                    .blocklist_type("__gnuc_va_list")+                    .blocklist_item("__GNUC_VA_LIST");+            }++            // handle __va_list_tag specially as it is referenced from sqlite3_api_routines+            // so if it is blocklisted, those references will be broken for loadable extensions.+            #[cfg(not(feature = "loadable_extension"))]+            {+                bindings = bindings.blocklist_type("__va_list_tag");+                // Remove once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+                #[cfg(feature = "loadable_extension")]+                {+                    bindings_with_comments = bindings_with_comments.blocklist_type("__va_list_tag");+                }+            }+            // when building as a loadable_extension, make __va_list_tag opaque instead of omitting it+            #[cfg(feature = "loadable_extension")]+            {+                bindings = bindings.opaque_type("__va_list_tag");+                // Remove once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+                #[cfg(feature = "loadable_extension")]+                {+                    bindings_with_comments = bindings_with_comments.opaque_type("__va_list_tag");+                }+            }+        }++        // rust-bindgen does not handle CPP macros that alias functions, so+        // when using sqlite3ext.h to support loadable extensions, the macros+        // that attempt to redefine sqlite3 API routines to be redirected through+        // the static instance of the sqlite3_api_routines structure do not result+        // in any code production.+        //+        // Before defining wrappers to take their place, we need to blocklist+        // all sqlite3 API functions since none of their symbols will be+        // available directly when being loaded as an extension.+        #[cfg(feature = "loadable_extension")]+        {+            bindings = bindings.blocklist_function(".*");+            // Remove once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+            #[cfg(feature = "loadable_extension")]+            {+                bindings_with_comments = bindings_with_comments.blocklist_function(".*");+            }         } +        // When building a loadable extension, make a copy of the bindgen+        // Builder so we can generate identical output with comments after we+        // generate the bindings without comments.+        //+        // Uncomment once bindgen can clone: https://github.com/rust-lang/rust-bindgen/issues/2132+        // #[cfg(feature = "loadable_extension")]+        // let mut bindings_with_comments = bindings.clone();++        #[cfg(feature = "loadable_extension")]+        {+            bindings_with_comments = bindings_with_comments+                .clang_arg("-fparse-all-comments")+                .generate_comments(true);+        }++        // Generate rust bindings (without comments)         bindings             .generate()             .unwrap_or_else(|_| panic!("could not run bindgen on header {}", header))             .write(Box::new(&mut output))             .expect("could not write output of bindgen");-        let mut output = String::from_utf8(output).expect("bindgen output was not UTF-8?!"); -        // rusqlite's functions feature ors in the SQLITE_DETERMINISTIC flag when it-        // can. This flag was added in SQLite 3.8.3, but oring it in in prior-        // versions of SQLite is harmless. We don't want to not build just-        // because this flag is missing (e.g., if we're linking against-        // SQLite 3.7.x), so append the flag manually if it isn't present in bindgen's-        // output.-        if !output.contains("pub const SQLITE_DETERMINISTIC") {-            output.push_str("\npub const SQLITE_DETERMINISTIC: i32 = 2048;\n");+        #[allow(unused_mut)]+        let mut output_string = String::from_utf8(output).expect("bindgen output was not UTF-8?!");++        #[cfg(feature = "loadable_extension")]+        {+            // When building a loadable extension, we need doc comments to be+            // included in the bindings we process to determine minimum version+            // requirements for each API function (i.e. those documented in+            // `sqlite3ext.h`)+            //+            // We could just add comments above, but then they would be included+            // in the generated bindings which increases their size by ~4x.+            //+            // Instead, we run bindgen again to generate another version of+            // the bindings that includes comments, use that to generate API+            // function wrappers, and then insert those wrappers into the+            // non-commented bindings generated above.+            let mut output_with_comments = Vec::new();+            bindings_with_comments+                .generate()+                .unwrap_or_else(|_| panic!("could not run bindgen on header {}", header))+                .write(Box::new(&mut output_with_comments))+                .expect("could not write output of bindgen with comments");+            let output_with_comments_string = String::from_utf8(output_with_comments)+                .expect("bindgen output with comments was not UTF-8?!");++            // Get the list of API functions supported by sqlite3_api_routines,+            // and add wrappers for each of the API functions to dispatch the API+            // call through the loadable_extension_sqlite3_api(), which is defined+            // in the crate outside the generated bindings.+            //+            // While parsing the bindings, we check for comments (in the form of doc+            // attributes) that contain sqlite3 version strings to generate runtime+            // checks for minimum version to prevent accessing elements beyond the+            // end of the sqlite3_api_routines struct when loaded into an older+            // sqlite.+            let api_routines_struct_name = "sqlite3_api_routines".to_owned();++            let api_routines_struct =+                match get_struct_by_name(&output_with_comments_string, &api_routines_struct_name) {+                    Some(s) => s,+                    None => {+                        panic!(+                            "Failed to find struct {} in early bindgen output",+                            &api_routines_struct_name+                        );+                    }+                };++            output_string.push_str(+                r#"++// The `loadable_extension_sqlite3_api` function is defined when compiled as a+// loadable_extension. It is used to safely access the static `SQLITE3_API`+// reference that is populated by a call to either`loadable_extension_init`+// or `loadable_extension_embedded_init`+use crate::loadable_extension_sqlite3_api;++// sqlite3 API wrappers to support loadable extensions (Note: these were generated from libsqlite3-sys/build.rs - not by rust-bindgen)++"#,+            );++            // compile a regex to match sqlite3 version strings that annotate+            // fields in the sqlite3_api_routines struct as comments in+            // the sqlite3ext.h sqlite header file+            let version_re = regex::Regex::new(r"[^0-9](?P<version>3[.][0-9]+[.][0-9]+)[^0-9]")+                .expect("failed to compile regex");++            // prior to any version comments, we don't enable any version checks+            // the earliest version mentioned is 3.3.13, and it appears from the+            // comments that nearly all of the earlier fields in+            // `sqlite3_api_routines` existed in the earliest version, with the+            // possible exception of `sqlite3_overload_function` which is+            // annotated with the comment `Added ???`+            //+            // I have added a special case for `sqlite3_overload_function`+            // to hard-code the version requirement for it to 3.3.13, which is+            // the fail-safe way of handling that ambiguity.+            let mut require_sqlite3_version = None;+            // create wrapper for each field in api routines struct+            for field in &api_routines_struct.fields {+                let ident = match &field.ident {+                    Some(ident) => ident,+                    None => {+                        panic!("Unexpected anonymous field in sqlite");+                    }+                };+                let field_type = &field.ty;+                for attr in &field.attrs {+                    match attr.path.get_ident() {+                        Some(ident) => {+                            if ident == "doc" {+                                for t in attr.tokens.clone().into_iter() {+                                    if let proc_macro2::TokenTree::Literal(l) = t {+                                        let literal_comments = l.to_string();+                                        if let Some(captures) =+                                            version_re.captures(&literal_comments)+                                        {+                                            if let Some(sqlite3_version_match) =+                                                captures.name("version")+                                            {+                                                let sqlite3_version =+                                                    sqlite3_version_match.as_str().to_string();+                                                match require_sqlite3_version {+                                                    None => {+                                                        require_sqlite3_version =+                                                            Some(sqlite3_version);+                                                    }+                                                    Some(require_sqlite3_version_str) => {+                                                        if version_compare::compare_to(+                                                            &sqlite3_version,+                                                            &require_sqlite3_version_str,+                                                            version_compare::Cmp::Lt,+                                                        )+                                                        .expect("failed to compare version strings")+                                                        {+                                                            panic!("Unexpectedly found sqlite3 version requirement in sqlite3ext.h that is lower than the version required for the previous field in sqlite3_api_routines (found '{}' after '{}')", &sqlite3_version, &require_sqlite3_version_str);+                                                        } else {+                                                            require_sqlite3_version =+                                                                Some(sqlite3_version);+                                                        }+                                                    }+                                                }+                                            } else {+                                                panic!("Regex matched but was missing version capture group");+                                            }+                                        }+                                    }+                                }+                            }+                        }+                        None => {}+                    }+                }++                // construct global sqlite api function identifier from field identifier+                let api_fn_name = format!("sqlite3_{}", ident);++                if (generating_bundled_bindings() || is_cross_compiling)+                    && blocklist_va_list_functions+                        .iter()+                        .any(|fn_name| *fn_name == api_fn_name)+                {+                    // skip this function as it is blocklisted when generating bundled bindings or cross compiling+                    continue;+                }++                // generate wrapper function and push it to output string+                let require_sqlite3_version_with_overrides =+                    if &api_fn_name == "sqlite3_overload_function" {+                        // override the version requirement for sqlite3_overload_function since it is commented+                        // with `Added ???` in sqlite3ext.h`+                        Some("3.3.13")+                    } else {+                        require_sqlite3_version.as_deref()+                    };+                let wrapper = generate_wrapper(+                    ident,+                    field_type,+                    &api_fn_name,+                    require_sqlite3_version_with_overrides,+                );+                output_string.push_str(&wrapper);+            }++            output_string.push('\n');         } +        #[allow(unused_mut)]         let mut file = OpenOptions::new()             .write(true)             .truncate(true)             .create(true)             .open(out_path)             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path)); -        file.write_all(output.as_bytes())+        #[cfg(not(feature = "loadable_extension"))]+        // the generated bindings have already been through rustfmt, just write them out+        file.write_all(output_string.as_bytes())             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path));+        #[cfg(feature = "loadable_extension")]+        write_with_rustfmt(file, output_string) // if we have generated loadable_extension bindings, pipe them through rustfmt as we write them out+            .unwrap_or_else(|e| panic!("Could not rustfmt output to {:?}: {:?}", out_path, e));+    }++    #[cfg(feature = "loadable_extension")]+    fn write_with_rustfmt(mut file: std::fs::File, output: String) -> Result<(), String> {+        // pipe generated bindings through rustfmt+        let rustfmt =+            which::which("rustfmt").map_err(|e| format!("rustfmt not on PATH: {:?}", e))?;+        let mut cmd = std::process::Command::new(rustfmt);+        cmd.stdin(std::process::Stdio::piped())+            .stdout(std::process::Stdio::piped());+        let mut rustfmt_child = cmd+            .spawn()+            .map_err(|e| format!("failed to execute rustfmt: {:?}", e))?;+        let mut rustfmt_child_stdin = rustfmt_child+            .stdin+            .take()+            .ok_or("failed to take rustfmt stdin")?;+        let mut rustfmt_child_stdout = rustfmt_child+            .stdout+            .take()+            .ok_or("failed to take rustfmt stdout")?;++        // spawn a thread to write output string to rustfmt stdin+        let stdin_handle = ::std::thread::spawn(move || {+            let _ = rustfmt_child_stdin.write_all(output.as_bytes());+            output+        });++        // read stdout of rustfmt and write it to bindings file at out_path+        std::io::copy(&mut rustfmt_child_stdout, &mut file)+            .map_err(|e| format!("failed to write to rustfmt stdin: {:?}", e))?;++        let status = rustfmt_child+            .wait()+            .map_err(|e| format!("failed to wait for rustfmt to complete: {:?}", e))?;+        stdin_handle+            .join()+            .map_err(|e| format!("unexpected error: failed to join rustfmt stdin: {:?}", e))?;++        match status.code() {+            Some(0) => {}+            Some(2) => {+                return Err("rustfmt parsing error".to_string());+            }+            Some(3) => {+                return Err("rustfmt could not format some lines.".to_string());+            }+            _ => {+                return Err("Internal rustfmt error".to_string());+            }+        }+        Ok(())+    }++    #[cfg(feature = "loadable_extension")]+    fn get_struct_by_name(bindgen_sources: &str, name: &str) -> Option<syn::ItemStruct> {+        let file = syn::parse_file(&bindgen_sources).expect("unable to parse early bindgen output");++        for item in &file.items {+            if let syn::Item::Struct(s) = item {+                if s.ident == name {+                    return Some(s.to_owned());+                }+            }+        }+        None+    }++    #[cfg(feature = "loadable_extension")]+    fn bare_fn_from_type_path(t: &syn::Type) -> syn::TypeBareFn {+        let path = match t {+            syn::Type::Path(tp) => &tp.path,+            _ => {+                panic!("type was not a type path");+            }+        };++        let mut path_args: Option<syn::PathArguments> = None;+        for segment in &path.segments {+            if segment.arguments.is_empty() {+                continue;+            }+            path_args = Some(segment.arguments.to_owned());+            break;+        }+        match path_args {+            Some(syn::PathArguments::AngleBracketed(p)) => {+                for gen_arg in p.args {+                    match gen_arg {+                        syn::GenericArgument::Type(syn::Type::BareFn(bf)) => {+                            return bf;+                        }+                        _ => {+                            panic!("parsed type was not a bare function as expected");+                        }+                    };+                }+            }+            _ => {+                panic!("parsed path args were not angle bracketed as expected");+            }+        };+        panic!("unexpected failure to parse bare function");+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_varargs_input_idents(+        field_ident: &syn::Ident,+        bare_fn: &syn::TypeBareFn,+        var_arg_types: &[&syn::Type],+    ) -> syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma> {+        use syn::Token;+        let mut api_fn_inputs = bare_fn.inputs.clone();+        for (index, var_arg_type) in var_arg_types.iter().enumerate() {+            let mut input = api_fn_inputs[api_fn_inputs.len() - 1].clone();+            let input_ident = syn::Ident::new(&format!("vararg{}", index + 1), field_ident.span());+            let colon = Token![:](field_ident.span());+            input.name = Some((input_ident, colon));+            input.ty = (*var_arg_type).to_owned();+            api_fn_inputs.push(input);+        }+        api_fn_inputs+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_wrapper(+        field_ident: &syn::Ident,+        syn_type: &syn::Type,+        api_fn_name: &str,+        require_sqlite3_version: Option<&str>,+    ) -> String {+        use quote::quote;+        use std::collections::BTreeMap;++        let field_name = field_ident.to_string();++        // add wrapper macro invocation to be appended to the generated bindings+        let bare_fn = bare_fn_from_type_path(syn_type);+        let api_fn_output = &bare_fn.output;++        // a map of wrapper function names to function inputs vectors+        let mut wrapper_fn_inputs_map: BTreeMap<+            String,+            syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma>,+        > = BTreeMap::new();++        // always generate a wrapper function of the same name as the api function name with no variadic arguments+        wrapper_fn_inputs_map.insert(+            api_fn_name.to_string(),+            generate_varargs_input_idents(field_ident, &bare_fn, &[]),+        );++        // handle variadic api functions by generating additional bindings for specific sets of method arguments that we support+        if bare_fn.variadic.is_some() {+            let const_c_char_type: syn::Type = syn::parse2(quote!(*const ::std::os::raw::c_char))+                .expect("failed to parse c_char type");+            let mut_void_type: syn::Type =+                syn::parse2(quote!(*mut ::core::ffi::c_void)).expect("failed to parse c_char type");+            let c_int_type: syn::Type =+                syn::parse2(quote!(::std::os::raw::c_int)).expect("failed to parse c_int type");+            let mut_c_int_type: syn::Type = syn::parse2(quote!(*mut ::std::os::raw::c_int))+                .expect("failed to parse mutable c_int reference");+            // until rust c_variadic support exists, we can't+            // transparently wrap variadic api functions.+            // generate specific set of args in place of+            // variadic for each function we care about.+            match api_fn_name {+                "sqlite3_db_config" => {+                    // https://sqlite.org/c3ref/c_dbconfig_defensive.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_constchar".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&const_c_char_type]),+                    ); // used for SQLITE_DBCONFIG_MAINDBNAME+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_void_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&mut_void_type, &c_int_type, &mut_c_int_type],+                        ),+                    ); // used for SQLITE_DBCONFIG_LOOKASIDE+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&c_int_type, &mut_c_int_type],+                        ),+                    ); // used for all other configuration verbs+                }+                "sqlite3_vtab_config" => {+                    // https://sqlite.org/c3ref/c_vtab_constraint_support.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_vtab_config_int".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&c_int_type]),+                    ); // used for SQLITE_VTAB_CONSTRAINT_SUPPORT+                }+                _ => {}+            };+        }++        let mut wrappers = String::new();+        for (api_fn_name, api_fn_inputs) in wrapper_fn_inputs_map {+            let api_fn_ident = syn::Ident::new(&api_fn_name, field_ident.span());++            // get identifiers for each of the inputs to use in the api call+            let api_fn_input_idents: Vec<syn::Ident> = (&api_fn_inputs)+                .into_iter()+                .map(|input| match &input.name {+                    Some((ident, _)) => ident.to_owned(),+                    _ => {+                        panic!("Input has no name {:#?}", input);+                    }+                })+                .collect();++            // Generate api version check to check the require_sqlite3_version_number+            // against the SQLITE_VERSION_NUMBER style integer version returned by a+            // sqlite3_libversion_number() call in order to protect api calls from+            // being made when the sqlite version the extension is loaded into does+            // not support them.+            // For more information, refer to sqlite docs: https://sqlite.org/c3ref/c_source_id.html+            let api_version_check_tokens = match require_sqlite3_version {+                Some(version) => {+                    let require_sqlite3_version = version_compare::Version::from(version)+                        .expect("failed to parse required sqlite3 version as Version");+                    let require_sqlite3_version_parts = require_sqlite3_version.parts();+                    if require_sqlite3_version_parts.len() != 3 {+                        panic!(+                            "sqlite3 version '{}' does not have exactly three parts",+                            version+                        );+                    }+                    let major = match require_sqlite3_version_parts[0] {+                        version_compare::Part::Number(major) => major,+                        _ => {+                            panic!(+                                "non-numeric major part found in sqlite3 version requirement '{}'",+                                require_sqlite3_version_parts[0]+                            );+                        }+                    };+                    let minor = match require_sqlite3_version_parts[1] {+                        version_compare::Part::Number(minor) => minor,+                        _ => {+                            panic!(+                                "non-numeric minor part found in sqlite3 version requirement '{}'",+                                require_sqlite3_version_parts[1]+                            );+                        }+                    };+                    let patch = match require_sqlite3_version_parts[2] {+                        version_compare::Part::Number(patch) => patch,+                        _ => {+                            panic!(+                                "non-numeric patch part found in sqlite3 version requirement '{}'",+                                require_sqlite3_version_parts[2]+                            );+                        }+                    };+                    let require_sqlite3_version_number = 1_000_000 * major + 1_000 * minor + patch;+                    quote! {+                        let sqlite3_version_number = sqlite3_libversion_number();+                        if sqlite3_version_number < #require_sqlite3_version_number {+                            panic!(stringify!("sqlite3 version is {} but version {} is required for the ", #api_fn_ident, " function"), sqlite3_version_number, #require_sqlite3_version_number);+                        }+                    }+                }+                None => quote! {},+            };++            // generate wrapper and return it as a string+            let wrapper_tokens = quote! {+                pub unsafe fn #api_fn_ident(#api_fn_inputs) #api_fn_output {+                    let p_api = loadable_extension_sqlite3_api();+                    #api_version_check_tokens+                    ((*p_api).#field_ident

My intent was that the code inserted by #api_version_check_tokens applies the runtime version check and prevents the next line from being executed if sqlite3_api_routines is smaller at runtime than at compile time.

I'm not extremely well versed in this part of Rust. I guess the undefined behaviour is because we are never allowed to make a reference (not a raw pointer, but a reference) to uninitialized data even if we don't execute it, because the compiler may rearrange execution order or something like that?

I'll make the change to raw pointers using addr_of and read - thanks for the suggestion!

jrandall

comment created time in a month

PullRequestReviewEvent

push eventGenomicsplc/rust-bindgen

Joshua C. Randall

commit sha 001c6b5073364429a2bd8f48d356fc2ab6f79315

add `builder-clone` feature to gate whether `Builder` implements `Clone` (and requires `ParseCallbacks` implementors to implement `Clone` as well)

view details

push time in a month

push eventGenomicsplc/rust-bindgen

Joshua C. Randall

commit sha fcdbef42a42a7493b53ca78931a9bdc40118b191

add `builder-clone` feature to gate whether `Builder` implements `Clone` (and requires `ParseCallbacks` implementors to implement `Clone` as well)

view details

push time in a month

pull request commentrust-lang/rust-bindgen

implement clone on builder

Spoke too soon - I now see that the bindgen-integration tests are failing due to MacroCallback not implementing Clone, and the use case there involves a HashSet and some Mutex-protected counters that don't really make sense to clone.

I suppose an option might be to put implementing Clone on Builder behind a feature gate so that it can be opted-in to when the ability to clone is desired and the need to implement Clone for any ParseCallbacks in use is trvial (or worth the effort).

I'll give that a try.

jrandall

comment created time in a month

push eventGenomicsplc/rust-bindgen

Joshua C. Randall

commit sha 576cd0fcfc3a1506506150cdbca6cba8fb58f85a

fix rustfmt issue in src/callbacks.rs

view details

push time in a month

PR opened rust-lang/rust-bindgen

implement clone on builder

Implements Clone on Builder (and all of the types included within it). Most were straightforward and I simply added derive Clone on the structs, except for the boxed ParseCallbacks since it is a trait. I have used the dyn-clone crate to make this work, provided that the underlying type implementing the trait implements Clone itself.

Notably, this does mean that any type used to implement ParseCallbacks that does not implement Clone would cause a compilation failure, although I imagine in most cases it should be a simple matter of adding #[derive(Clone)] to the type. Every example I've seen uses a unit struct for the ParseCallbacks implementation, so I guess it would be very easy to fix, but it would nonetheless be a breaking change since code that previously worked would fail to compile after this change until a Clone implementation is added to the type implementing ParseCallbacks.

The changes made to the parse callbacks tests illustrate the simple fix that would need to be made to client code: https://github.com/rust-lang/rust-bindgen/commit/97fed5531db5ec2e55426f2fe2262e1c3de0d40c

Closes #2132

+19 -8

0 comment

8 changed files

pr created time in a month

push eventGenomicsplc/rust-bindgen

Joshua C. Randall

commit sha 97fed5531db5ec2e55426f2fe2262e1c3de0d40c

derive Clone for structs in ParseCallbacks tests

view details

push time in a month

create barnchGenomicsplc/rust-bindgen

branch : 2132-implement-clone-on-builder

created branch time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 92fa7030ba5b3a498f404405b903f19eff03ca39

add support for sqlite loadable extensions

view details

Joshua C. Randall

commit sha 5c97c48fe0a08a8e41db528ae411b1dcb2653982

install rustfmt in travis ci

view details

Joshua C. Randall

commit sha 4caff1491f4c380a33c4adc66025423d58d437e8

add rustfmt to appveyor install

view details

Joshua C. Randall

commit sha b21aed20f037dbc3b2cff06bf4262ba1ff488099

fix reference to link_lib in vcpkg build

view details

gwenn

commit sha 0bf1e742726e70202e9d702328b4b11f8880a7d9

Merge remote-tracking branch 'upstream/master' into pr/loadable-extensions # Conflicts: # Cargo.toml # libsqlite3-sys/Cargo.toml # libsqlite3-sys/build.rs # src/inner_connection.rs # src/lib.rs

view details

gwenn

commit sha 955521fa243b56ce74ff7eefd7dfc9b91447c579

Remove old bindgens

view details

gwenn

commit sha 26c9193fa3e2830292f0ec925aafd031795673b8

Remove some more old bindgens

view details

gwenn

commit sha e1fffce7636e30b44b4b28e383687d299bd00e5b

Remove unused `api_routines_stub` feature

view details

gwenn

commit sha d4b710a4f4b09699720109c1a34ebc4faf1893ea

Remove unused `vtab_v3` feature

view details

gwenn

commit sha 2b3fd57236dd52d7ef9a2c9443ff8844f542ad6c

Merge remote-tracking branch 'upstream/master' into pr/loadable-extensions

view details

gwenn

commit sha 168821a19dbdf9b157ba87cc0b13f44d2022e92f

Revert changes on old bindgens

view details

gwenn

commit sha 2c6b6acb6a4878f16d4b52f79ba62512dcb87d22

Ignore PATH change on Windows

view details

gwenn

commit sha 68bea40e806df992d11ff8d12480738fc34bb611

Remove copyright header from build script

view details

gwenn

commit sha 325bfb6344fe94c595a34ab79cdfba6aa0e12d21

Generate all three bindgen files

view details

gwenn

commit sha 2b170f0e1612484a06c0370d7838cc60ef47e841

Generate new bindgen files for bundled version

view details

gwenn

commit sha 3494e67f3c651202db033dae740ee652c4550dfd

Remove generate-bindgen-bindings script If we need to regenerate old bindings, we can still do it with upgrade.sh

view details

gwenn

commit sha 7dd7204aa519308d9c4bde405fca815938ab1717

Upgrade syn / quote

view details

gwenn

commit sha 1e19268d2bdf52f0bf16e118544ba5c3f3019197

Fix warning after wrong merge

view details

gwenn

commit sha 0b042d061aeec3ca7398be7f8eae343b1778390e

Run rustfmt manually only while regenerating bindgen files

view details

gwenn

commit sha 9be9358c1a5bf88b7bcac19a24725b209aa29b85

Fix warnings in build script

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha f3c8b2934a8d3368bef82e9f896341e2d23aeeeb

cargo fmt

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha aa52f56cc47638348c844edefc60d6a2e7be5a55

cargo fmt

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 069fc87fa12d5ddd220d1550618f0767ec403b39

cargo fmt

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 2472ec98c89c9db4a66e59625054ebabb2b83d30

remove extraneous cmoment line in libsqlite3-sys/build.rs

view details

Joshua C. Randall

commit sha 50694746dc9abaf54ef237d61ac75f2bc7eefc10

add runtime version checks to loadable extension wrappers, improve safety, and add explicit entry points for both embedded and non-embedded loadable extensions

view details

Joshua C. Randall

commit sha dd40fe05ee096383589c7b3abcfa74c7d59c1813

cargo fmt

view details

Joshua C. Randall

commit sha 7e020f0982db552590c08186dcd48cb873587cf1

add safety docs to unsafe functions in example-extension and example-embedded-extension

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha b252f5e88e8452ba567930899d09df0fff5e1910

cargo fmt

view details

Joshua C. Randall

commit sha 55c0c3cfcc350a06b69256c4b2ca41d30a03824f

add safety docs to unsafe functions in example-extension and example-embedded-extension

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 22e273924b983288282924f3e28be467ccce6b3b

cargo fmt

view details

Joshua C. Randall

commit sha 5135abe011880b88aa49bdaaaaa89ee2cdc78435

add safety docs to unsafe functions in example-extension and example-embedded-extension

view details

push time in a month

push eventGenomicsplc/rusqlite

Joshua C. Randall

commit sha 71b022ce6b786d6c5ed21582bc0889bcff0e0ee2

Merge

view details

push time in a month

PullRequestReviewEvent

Pull request review commentrusqlite/rusqlite

Add support for sqlite loadable extensions

 mod bindings {         let target_arch = std::env::var("TARGET").unwrap();         let host_arch = std::env::var("HOST").unwrap();         let is_cross_compiling = target_arch != host_arch;-+        let blocklist_va_list_functions = &vec![+            "sqlite3_vmprintf",+            "sqlite3_vsnprintf",+            "sqlite3_xvsnprintf",+            "sqlite3_str_vappendf",+        ];         // Note that when generating the bundled file, we're essentially always         // cross compiling.         if generating_bundled_bindings() || is_cross_compiling {-            // Get rid of va_list, as it's not+            // get rid of blocklisted functions that use va_list+            for fn_name in blocklist_va_list_functions {+                bindings = bindings.blocklist_function(fn_name)+            }+            // Get rid of va_list             bindings = bindings-                .blocklist_function("sqlite3_vmprintf")-                .blocklist_function("sqlite3_vsnprintf")-                .blocklist_function("sqlite3_str_vappendf")                 .blocklist_type("va_list")                 .blocklist_type("__builtin_va_list")                 .blocklist_type("__gnuc_va_list")-                .blocklist_type("__va_list_tag")                 .blocklist_item("__GNUC_VA_LIST");++            // handle __va_list_tag specially as it is referenced from sqlite3_api_routines+            // so if it is blocklisted, those references will be broken for loadable extensions.+            #[cfg(not(feature = "loadable_extension"))]+            {+                bindings = bindings.blocklist_type("__va_list_tag");+            }+            // when building as a loadable_extension, make __va_list_tag opaque instead of omitting it+            #[cfg(feature = "loadable_extension")]+            {+                bindings = bindings.opaque_type("__va_list_tag");+            }+        }++        // rust-bindgen does not handle CPP macros that alias functions, so+        // when using sqlite3ext.h to support loadable extensions, the macros+        // that attempt to redefine sqlite3 API routines to be redirected through+        // the global sqlite3_api instance of the sqlite3_api_routines structure+        // do not result in any code production.+        //+        // Before defining wrappers to take their place, we need to blocklist+        // all sqlite3 API functions since none of their symbols will be+        // available directly when being loaded as an extension.+        #[cfg(feature = "loadable_extension")]+        {+            // some api functions do not have an implementation in sqlite3_api_routines+            // (for example: sqlite3_config, sqlite3_initialize, sqlite3_interrupt, ...).+            // while this isn't a problem for shared libraries (unless we actually try to+            // call them, it is better to blocklist them all so that the build will fail+            // if an attempt is made to call an extern function that we know won't exist+            // and to avoid undefined symbol issues when linking the loadable extension+            // rust code with other (e.g. non-rust) code+            bindings = bindings.blocklist_function(".*")         }          bindings             .generate()             .unwrap_or_else(|_| panic!("could not run bindgen on header {}", header))             .write(Box::new(&mut output))             .expect("could not write output of bindgen");-        let mut output = String::from_utf8(output).expect("bindgen output was not UTF-8?!"); -        // rusqlite's functions feature ors in the SQLITE_DETERMINISTIC flag when it-        // can. This flag was added in SQLite 3.8.3, but oring it in in prior-        // versions of SQLite is harmless. We don't want to not build just-        // because this flag is missing (e.g., if we're linking against-        // SQLite 3.7.x), so append the flag manually if it isn't present in bindgen's-        // output.-        if !output.contains("pub const SQLITE_DETERMINISTIC") {-            output.push_str("\npub const SQLITE_DETERMINISTIC: i32 = 2048;\n");+        #[allow(unused_mut)]+        let mut output_string = String::from_utf8(output).expect("bindgen output was not UTF-8?!");++        // Get the list of API functions supported by sqlite3_api_routines,+        // set the corresponding sqlite3 api routine to be blocklisted in the+        // final bindgen run, and add wrappers for each of the API functions to+        // dispatch the API call through a sqlite3_api global, which is defined+        // outside the generated bindings in lib.rs, either as a built-in static+        // or an extern symbol in the case of loadable_extension_embedded (i.e.+        // when the rust code will be a part of an extension but not implement+        // the extension entrypoint itself).+        #[cfg(feature = "loadable_extension")]+        {+            let api_routines_struct_name = "sqlite3_api_routines".to_owned();++            let api_routines_struct =+                match get_struct_by_name(&output_string, &api_routines_struct_name) {+                    Some(s) => s,+                    None => {+                        panic!(+                            "Failed to find struct {} in early bindgen output",+                            &api_routines_struct_name+                        );+                    }+                };++            output_string.push_str(+                r#"++// sqlite3_api is defined in lib.rs as either a static or an extern when compiled as a loadable_extension+use crate::sqlite3_api;++// sqlite3 API wrappers to support loadable extensions (Note: these were generated from build.rs - not by rust-bindgen)++"#,+            );++            // create wrapper for each field in api routines struct+            for field in &api_routines_struct.fields {+                let ident = match &field.ident {+                    Some(ident) => ident,+                    None => {+                        panic!("Unexpected anonymous field in sqlite");+                    }+                };+                let field_type = &field.ty;++                // construct global sqlite api function identifier from field identifier+                let api_fn_name = format!("sqlite3_{}", ident);++                if (generating_bundled_bindings() || is_cross_compiling)+                    && blocklist_va_list_functions+                        .iter()+                        .any(|fn_name| *fn_name == api_fn_name)+                {+                    // skip this function as it is blocklisted when generating bundled bindings or cross compiling+                    continue;+                }++                // generate wrapper function and push it to output string+                let wrapper = generate_wrapper(ident, field_type, &api_fn_name);+                output_string.push_str(&wrapper);+            }++            output_string.push('\n');         } +        #[allow(unused_mut)]         let mut file = OpenOptions::new()             .write(true)             .truncate(true)             .create(true)             .open(out_path)             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path)); -        file.write_all(output.as_bytes())+        #[cfg(not(feature = "loadable_extension"))]+        // the generated bindings have already been through rustfmt, just write them out+        file.write_all(output_string.as_bytes())             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path));+        #[cfg(feature = "loadable_extension")]+        write_with_rustfmt(file, output_string) // if we have generated loadable_extension bindings, pipe them through rustfmt as we write them out+            .unwrap_or_else(|e| panic!("Could not rustfmt output to {:?}: {:?}", out_path, e));+    }++    #[cfg(feature = "loadable_extension")]+    fn write_with_rustfmt(mut file: std::fs::File, output: String) -> Result<(), String> {+        // pipe generated bindings through rustfmt+        let rustfmt =+            which::which("rustfmt").map_err(|e| format!("rustfmt not on PATH: {:?}", e))?;+        let mut cmd = std::process::Command::new(rustfmt);+        cmd.stdin(std::process::Stdio::piped())+            .stdout(std::process::Stdio::piped());+        let mut rustfmt_child = cmd+            .spawn()+            .map_err(|e| format!("failed to execute rustfmt: {:?}", e))?;+        let mut rustfmt_child_stdin = rustfmt_child+            .stdin+            .take()+            .ok_or("failed to take rustfmt stdin")?;+        let mut rustfmt_child_stdout = rustfmt_child+            .stdout+            .take()+            .ok_or("failed to take rustfmt stdout")?;++        // spawn a thread to write output string to rustfmt stdin+        let stdin_handle = ::std::thread::spawn(move || {+            let _ = rustfmt_child_stdin.write_all(output.as_bytes());+            output+        });++        // read stdout of rustfmt and write it to bindings file at out_path+        std::io::copy(&mut rustfmt_child_stdout, &mut file)+            .map_err(|e| format!("failed to write to rustfmt stdin: {:?}", e))?;++        let status = rustfmt_child+            .wait()+            .map_err(|e| format!("failed to wait for rustfmt to complete: {:?}", e))?;+        stdin_handle+            .join()+            .map_err(|e| format!("unexpected error: failed to join rustfmt stdin: {:?}", e))?;++        match status.code() {+            Some(0) => {}+            Some(2) => {+                return Err("rustfmt parsing error".to_string());+            }+            Some(3) => {+                return Err("rustfmt could not format some lines.".to_string());+            }+            _ => {+                return Err("Internal rustfmt error".to_string());+            }+        }+        Ok(())+    }++    #[cfg(feature = "loadable_extension")]+    fn get_struct_by_name(bindgen_sources: &str, name: &str) -> Option<syn::ItemStruct> {+        let file = syn::parse_file(&bindgen_sources).expect("unable to parse early bindgen output");++        for item in &file.items {+            if let syn::Item::Struct(s) = item {+                if s.ident == name {+                    return Some(s.to_owned());+                }+            }+        }+        None+    }++    #[cfg(feature = "loadable_extension")]+    fn bare_fn_from_type_path(t: &syn::Type) -> syn::TypeBareFn {+        let path = match t {+            syn::Type::Path(tp) => &tp.path,+            _ => {+                panic!("type was not a type path");+            }+        };++        let mut path_args: Option<syn::PathArguments> = None;+        for segment in &path.segments {+            if segment.arguments.is_empty() {+                continue;+            }+            path_args = Some(segment.arguments.to_owned());+            break;+        }+        match path_args {+            Some(syn::PathArguments::AngleBracketed(p)) => {+                for gen_arg in p.args {+                    match gen_arg {+                        syn::GenericArgument::Type(syn::Type::BareFn(bf)) => {+                            return bf;+                        }+                        _ => {+                            panic!("parsed type was not a bare function as expected");+                        }+                    };+                }+            }+            _ => {+                panic!("parsed path args were not angle bracketed as expected");+            }+        };+        panic!("unexpected failure to parse bare function");+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_varargs_input_idents(+        field_ident: &syn::Ident,+        bare_fn: &syn::TypeBareFn,+        var_arg_types: &[&syn::Type],+    ) -> syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma> {+        use syn::Token;+        let mut api_fn_inputs = bare_fn.inputs.clone();+        for (index, var_arg_type) in var_arg_types.iter().enumerate() {+            let mut input = api_fn_inputs[api_fn_inputs.len() - 1].clone();+            let input_ident = syn::Ident::new(&format!("vararg{}", index + 1), field_ident.span());+            let colon = Token![:](field_ident.span());+            input.name = Some((input_ident, colon));+            input.ty = (*var_arg_type).to_owned();+            api_fn_inputs.push(input);+        }+        api_fn_inputs+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_wrapper(+        field_ident: &syn::Ident,+        syn_type: &syn::Type,+        api_fn_name: &str,+    ) -> String {+        use quote::quote;+        use std::collections::BTreeMap;++        let field_name = field_ident.to_string();++        // add wrapper macro invocation to be appended to the generated bindings+        let bare_fn = bare_fn_from_type_path(syn_type);+        let api_fn_output = &bare_fn.output;++        // a map of wrapper function names to function inputs vectors+        let mut wrapper_fn_inputs_map: BTreeMap<+            String,+            syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma>,+        > = BTreeMap::new();++        // always generate a wrapper function of the same name as the api function name with no variadic arguments+        wrapper_fn_inputs_map.insert(+            api_fn_name.to_string(),+            generate_varargs_input_idents(field_ident, &bare_fn, &[]),+        );++        // handle variadic api functions by generating additional bindings for specific sets of method arguments that we support+        if bare_fn.variadic.is_some() {+            let const_c_char_type: syn::Type = syn::parse2(quote!(*const ::std::os::raw::c_char))+                .expect("failed to parse c_char type");+            let mut_void_type: syn::Type =+                syn::parse2(quote!(*mut ::core::ffi::c_void)).expect("failed to parse c_char type");+            let c_int_type: syn::Type =+                syn::parse2(quote!(::std::os::raw::c_int)).expect("failed to parse c_int type");+            let mut_c_int_type: syn::Type = syn::parse2(quote!(*mut ::std::os::raw::c_int))+                .expect("failed to parse mutable c_int reference");+            // until rust c_variadic support exists, we can't+            // transparently wrap variadic api functions.+            // generate specific set of args in place of+            // variadic for each function we care about.+            match api_fn_name {+                "sqlite3_db_config" => {+                    // https://sqlite.org/c3ref/c_dbconfig_defensive.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_constchar".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&const_c_char_type]),+                    ); // used for SQLITE_DBCONFIG_MAINDBNAME+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_void_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&mut_void_type, &c_int_type, &mut_c_int_type],+                        ),+                    ); // used for SQLITE_DBCONFIG_LOOKASIDE+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&c_int_type, &mut_c_int_type],+                        ),+                    ); // used for all other configuration verbs+                }+                "sqlite3_vtab_config" => {+                    // https://sqlite.org/c3ref/c_vtab_constraint_support.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_vtab_config_int".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&c_int_type]),+                    ); // used for SQLITE_VTAB_CONSTRAINT_SUPPORT+                }+                _ => {}+            };+        }++        let mut wrappers = String::new();+        for (api_fn_name, api_fn_inputs) in wrapper_fn_inputs_map {+            let api_fn_ident = syn::Ident::new(&api_fn_name, field_ident.span());++            // get identifiers for each of the inputs to use in the api call+            let api_fn_input_idents: Vec<syn::Ident> = (&api_fn_inputs)+                .into_iter()+                .map(|input| match &input.name {+                    Some((ident, _)) => ident.to_owned(),+                    _ => {+                        panic!("Input has no name {:#?}", input);+                    }+                })+                .collect();++            // generate wrapper and return it as a string+            let wrapper_tokens = quote! {+                pub unsafe fn #api_fn_ident(#api_fn_inputs) #api_fn_output {+                    if sqlite3_api.is_null() {+                        panic!("sqlite3_api is null");+                    }+                    ((*sqlite3_api).#field_ident+                        .expect(stringify!("sqlite3_api contains null pointer for ", #field_name, " function")))(+                            #(#api_fn_input_idents),*+                    )+                }

Runtime version checking has now been implemented by generating a separate set of rust-bindgen bindings from sqlite3ext.h including comments and parsing the sqlite_api_routine fields along with the inline version comments that indicate minimum versions for each section of added fields.

jrandall

comment created time in a month

Pull request review commentrusqlite/rusqlite

Add support for sqlite loadable extensions

 mod bindings {         let target_arch = std::env::var("TARGET").unwrap();         let host_arch = std::env::var("HOST").unwrap();         let is_cross_compiling = target_arch != host_arch;-+        let blocklist_va_list_functions = &vec![+            "sqlite3_vmprintf",+            "sqlite3_vsnprintf",+            "sqlite3_xvsnprintf",+            "sqlite3_str_vappendf",+        ];         // Note that when generating the bundled file, we're essentially always         // cross compiling.         if generating_bundled_bindings() || is_cross_compiling {-            // Get rid of va_list, as it's not+            // get rid of blocklisted functions that use va_list+            for fn_name in blocklist_va_list_functions {+                bindings = bindings.blocklist_function(fn_name)+            }+            // Get rid of va_list             bindings = bindings-                .blocklist_function("sqlite3_vmprintf")-                .blocklist_function("sqlite3_vsnprintf")-                .blocklist_function("sqlite3_str_vappendf")                 .blocklist_type("va_list")                 .blocklist_type("__builtin_va_list")                 .blocklist_type("__gnuc_va_list")-                .blocklist_type("__va_list_tag")                 .blocklist_item("__GNUC_VA_LIST");++            // handle __va_list_tag specially as it is referenced from sqlite3_api_routines+            // so if it is blocklisted, those references will be broken for loadable extensions.+            #[cfg(not(feature = "loadable_extension"))]+            {+                bindings = bindings.blocklist_type("__va_list_tag");+            }+            // when building as a loadable_extension, make __va_list_tag opaque instead of omitting it+            #[cfg(feature = "loadable_extension")]+            {+                bindings = bindings.opaque_type("__va_list_tag");+            }+        }++        // rust-bindgen does not handle CPP macros that alias functions, so+        // when using sqlite3ext.h to support loadable extensions, the macros+        // that attempt to redefine sqlite3 API routines to be redirected through+        // the global sqlite3_api instance of the sqlite3_api_routines structure+        // do not result in any code production.+        //+        // Before defining wrappers to take their place, we need to blocklist+        // all sqlite3 API functions since none of their symbols will be+        // available directly when being loaded as an extension.+        #[cfg(feature = "loadable_extension")]+        {+            // some api functions do not have an implementation in sqlite3_api_routines+            // (for example: sqlite3_config, sqlite3_initialize, sqlite3_interrupt, ...).+            // while this isn't a problem for shared libraries (unless we actually try to+            // call them, it is better to blocklist them all so that the build will fail+            // if an attempt is made to call an extern function that we know won't exist+            // and to avoid undefined symbol issues when linking the loadable extension+            // rust code with other (e.g. non-rust) code+            bindings = bindings.blocklist_function(".*")         }          bindings             .generate()             .unwrap_or_else(|_| panic!("could not run bindgen on header {}", header))             .write(Box::new(&mut output))             .expect("could not write output of bindgen");-        let mut output = String::from_utf8(output).expect("bindgen output was not UTF-8?!"); -        // rusqlite's functions feature ors in the SQLITE_DETERMINISTIC flag when it-        // can. This flag was added in SQLite 3.8.3, but oring it in in prior-        // versions of SQLite is harmless. We don't want to not build just-        // because this flag is missing (e.g., if we're linking against-        // SQLite 3.7.x), so append the flag manually if it isn't present in bindgen's-        // output.-        if !output.contains("pub const SQLITE_DETERMINISTIC") {-            output.push_str("\npub const SQLITE_DETERMINISTIC: i32 = 2048;\n");+        #[allow(unused_mut)]+        let mut output_string = String::from_utf8(output).expect("bindgen output was not UTF-8?!");++        // Get the list of API functions supported by sqlite3_api_routines,+        // set the corresponding sqlite3 api routine to be blocklisted in the+        // final bindgen run, and add wrappers for each of the API functions to+        // dispatch the API call through a sqlite3_api global, which is defined+        // outside the generated bindings in lib.rs, either as a built-in static+        // or an extern symbol in the case of loadable_extension_embedded (i.e.+        // when the rust code will be a part of an extension but not implement+        // the extension entrypoint itself).+        #[cfg(feature = "loadable_extension")]+        {+            let api_routines_struct_name = "sqlite3_api_routines".to_owned();++            let api_routines_struct =+                match get_struct_by_name(&output_string, &api_routines_struct_name) {+                    Some(s) => s,+                    None => {+                        panic!(+                            "Failed to find struct {} in early bindgen output",+                            &api_routines_struct_name+                        );+                    }+                };++            output_string.push_str(+                r#"++// sqlite3_api is defined in lib.rs as either a static or an extern when compiled as a loadable_extension+use crate::sqlite3_api;++// sqlite3 API wrappers to support loadable extensions (Note: these were generated from build.rs - not by rust-bindgen)++"#,+            );++            // create wrapper for each field in api routines struct+            for field in &api_routines_struct.fields {+                let ident = match &field.ident {+                    Some(ident) => ident,+                    None => {+                        panic!("Unexpected anonymous field in sqlite");+                    }+                };+                let field_type = &field.ty;++                // construct global sqlite api function identifier from field identifier+                let api_fn_name = format!("sqlite3_{}", ident);++                if (generating_bundled_bindings() || is_cross_compiling)+                    && blocklist_va_list_functions+                        .iter()+                        .any(|fn_name| *fn_name == api_fn_name)+                {+                    // skip this function as it is blocklisted when generating bundled bindings or cross compiling+                    continue;+                }++                // generate wrapper function and push it to output string+                let wrapper = generate_wrapper(ident, field_type, &api_fn_name);+                output_string.push_str(&wrapper);+            }++            output_string.push('\n');         } +        #[allow(unused_mut)]         let mut file = OpenOptions::new()             .write(true)             .truncate(true)             .create(true)             .open(out_path)             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path)); -        file.write_all(output.as_bytes())+        #[cfg(not(feature = "loadable_extension"))]+        // the generated bindings have already been through rustfmt, just write them out+        file.write_all(output_string.as_bytes())             .unwrap_or_else(|_| panic!("Could not write to {:?}", out_path));+        #[cfg(feature = "loadable_extension")]+        write_with_rustfmt(file, output_string) // if we have generated loadable_extension bindings, pipe them through rustfmt as we write them out+            .unwrap_or_else(|e| panic!("Could not rustfmt output to {:?}: {:?}", out_path, e));+    }++    #[cfg(feature = "loadable_extension")]+    fn write_with_rustfmt(mut file: std::fs::File, output: String) -> Result<(), String> {+        // pipe generated bindings through rustfmt+        let rustfmt =+            which::which("rustfmt").map_err(|e| format!("rustfmt not on PATH: {:?}", e))?;+        let mut cmd = std::process::Command::new(rustfmt);+        cmd.stdin(std::process::Stdio::piped())+            .stdout(std::process::Stdio::piped());+        let mut rustfmt_child = cmd+            .spawn()+            .map_err(|e| format!("failed to execute rustfmt: {:?}", e))?;+        let mut rustfmt_child_stdin = rustfmt_child+            .stdin+            .take()+            .ok_or("failed to take rustfmt stdin")?;+        let mut rustfmt_child_stdout = rustfmt_child+            .stdout+            .take()+            .ok_or("failed to take rustfmt stdout")?;++        // spawn a thread to write output string to rustfmt stdin+        let stdin_handle = ::std::thread::spawn(move || {+            let _ = rustfmt_child_stdin.write_all(output.as_bytes());+            output+        });++        // read stdout of rustfmt and write it to bindings file at out_path+        std::io::copy(&mut rustfmt_child_stdout, &mut file)+            .map_err(|e| format!("failed to write to rustfmt stdin: {:?}", e))?;++        let status = rustfmt_child+            .wait()+            .map_err(|e| format!("failed to wait for rustfmt to complete: {:?}", e))?;+        stdin_handle+            .join()+            .map_err(|e| format!("unexpected error: failed to join rustfmt stdin: {:?}", e))?;++        match status.code() {+            Some(0) => {}+            Some(2) => {+                return Err("rustfmt parsing error".to_string());+            }+            Some(3) => {+                return Err("rustfmt could not format some lines.".to_string());+            }+            _ => {+                return Err("Internal rustfmt error".to_string());+            }+        }+        Ok(())+    }++    #[cfg(feature = "loadable_extension")]+    fn get_struct_by_name(bindgen_sources: &str, name: &str) -> Option<syn::ItemStruct> {+        let file = syn::parse_file(&bindgen_sources).expect("unable to parse early bindgen output");++        for item in &file.items {+            if let syn::Item::Struct(s) = item {+                if s.ident == name {+                    return Some(s.to_owned());+                }+            }+        }+        None+    }++    #[cfg(feature = "loadable_extension")]+    fn bare_fn_from_type_path(t: &syn::Type) -> syn::TypeBareFn {+        let path = match t {+            syn::Type::Path(tp) => &tp.path,+            _ => {+                panic!("type was not a type path");+            }+        };++        let mut path_args: Option<syn::PathArguments> = None;+        for segment in &path.segments {+            if segment.arguments.is_empty() {+                continue;+            }+            path_args = Some(segment.arguments.to_owned());+            break;+        }+        match path_args {+            Some(syn::PathArguments::AngleBracketed(p)) => {+                for gen_arg in p.args {+                    match gen_arg {+                        syn::GenericArgument::Type(syn::Type::BareFn(bf)) => {+                            return bf;+                        }+                        _ => {+                            panic!("parsed type was not a bare function as expected");+                        }+                    };+                }+            }+            _ => {+                panic!("parsed path args were not angle bracketed as expected");+            }+        };+        panic!("unexpected failure to parse bare function");+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_varargs_input_idents(+        field_ident: &syn::Ident,+        bare_fn: &syn::TypeBareFn,+        var_arg_types: &[&syn::Type],+    ) -> syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma> {+        use syn::Token;+        let mut api_fn_inputs = bare_fn.inputs.clone();+        for (index, var_arg_type) in var_arg_types.iter().enumerate() {+            let mut input = api_fn_inputs[api_fn_inputs.len() - 1].clone();+            let input_ident = syn::Ident::new(&format!("vararg{}", index + 1), field_ident.span());+            let colon = Token![:](field_ident.span());+            input.name = Some((input_ident, colon));+            input.ty = (*var_arg_type).to_owned();+            api_fn_inputs.push(input);+        }+        api_fn_inputs+    }++    #[cfg(feature = "loadable_extension")]+    fn generate_wrapper(+        field_ident: &syn::Ident,+        syn_type: &syn::Type,+        api_fn_name: &str,+    ) -> String {+        use quote::quote;+        use std::collections::BTreeMap;++        let field_name = field_ident.to_string();++        // add wrapper macro invocation to be appended to the generated bindings+        let bare_fn = bare_fn_from_type_path(syn_type);+        let api_fn_output = &bare_fn.output;++        // a map of wrapper function names to function inputs vectors+        let mut wrapper_fn_inputs_map: BTreeMap<+            String,+            syn::punctuated::Punctuated<syn::BareFnArg, syn::token::Comma>,+        > = BTreeMap::new();++        // always generate a wrapper function of the same name as the api function name with no variadic arguments+        wrapper_fn_inputs_map.insert(+            api_fn_name.to_string(),+            generate_varargs_input_idents(field_ident, &bare_fn, &[]),+        );++        // handle variadic api functions by generating additional bindings for specific sets of method arguments that we support+        if bare_fn.variadic.is_some() {+            let const_c_char_type: syn::Type = syn::parse2(quote!(*const ::std::os::raw::c_char))+                .expect("failed to parse c_char type");+            let mut_void_type: syn::Type =+                syn::parse2(quote!(*mut ::core::ffi::c_void)).expect("failed to parse c_char type");+            let c_int_type: syn::Type =+                syn::parse2(quote!(::std::os::raw::c_int)).expect("failed to parse c_int type");+            let mut_c_int_type: syn::Type = syn::parse2(quote!(*mut ::std::os::raw::c_int))+                .expect("failed to parse mutable c_int reference");+            // until rust c_variadic support exists, we can't+            // transparently wrap variadic api functions.+            // generate specific set of args in place of+            // variadic for each function we care about.+            match api_fn_name {+                "sqlite3_db_config" => {+                    // https://sqlite.org/c3ref/c_dbconfig_defensive.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_constchar".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&const_c_char_type]),+                    ); // used for SQLITE_DBCONFIG_MAINDBNAME+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_void_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&mut_void_type, &c_int_type, &mut_c_int_type],+                        ),+                    ); // used for SQLITE_DBCONFIG_LOOKASIDE+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_db_config_int_mutint".to_string(),+                        generate_varargs_input_idents(+                            field_ident,+                            &bare_fn,+                            &[&c_int_type, &mut_c_int_type],+                        ),+                    ); // used for all other configuration verbs+                }+                "sqlite3_vtab_config" => {+                    // https://sqlite.org/c3ref/c_vtab_constraint_support.html+                    wrapper_fn_inputs_map.insert(+                        "sqlite3_vtab_config_int".to_string(),+                        generate_varargs_input_idents(field_ident, &bare_fn, &[&c_int_type]),+                    ); // used for SQLITE_VTAB_CONSTRAINT_SUPPORT+                }+                _ => {}+            };+        }

Don't currently have a proposal to address this. At the moment we are only implementing calls to a small number of variadic API functions (that I am actually using).

@thomcc would you be happier if inclusion of these bindings was behind a feature? Something like experimental_variadic_support?

jrandall

comment created time in a month

PullRequestReviewEvent

Pull request review commentrusqlite/rusqlite

Add support for sqlite loadable extensions

+use std::marker::PhantomData;+use std::os::raw::{c_char, c_int};++use rusqlite::vtab::{+    eponymous_only_module, sqlite3_vtab, sqlite3_vtab_cursor, Context, IndexInfo, VTab,+    VTabConnection, VTabCursor, Values,+};+use rusqlite::{+    ffi,+    functions::FunctionFlags,+    types::{ToSqlOutput, Value},+};+use rusqlite::{to_sqlite_error, Connection, Result};++#[allow(clippy::not_unsafe_ptr_arg_deref)]+#[no_mangle]+pub extern "C" fn sqlite3_extension_init(

fixed

jrandall

comment created time in a month

PullRequestReviewEvent
more