Skip to content

CAY-3056 Switch DataMap queries to the new query parser #655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stariy95 wants to merge 2 commits into
base: master
Choose a base branch
from
+3,125 −1,668
Open

CAY-3056 Switch DataMap queries to the new query parser #655

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3a57ddb
CAY-3056 Switch DataMap queries to the new query parser
stariy95 Oct 23, 2026
0517dec
CAY-3056 Switch DataMap queries to the new query parser
stariy95 Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RELEASE-NOTES.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ CAY-3049 Cheap ClockMap to replace ConcurrentLinkedHashMap
CAY-3050 Array-based DataRows
CAY-3051 Replace PropertyVisitor with ClassDescriptor property collections
CAY-3054 Remove ObjRelationship.isReadOnly()
CAY-3056 Switch DataMap queries to the new query parser
CAY-3057 Improve parameter binding in the query parser

Bug Fixes:
Expand Down
8 changes: 8 additions & 0 deletions UPGRADE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,14 @@ List<Artist> artists = ObjectSelect
`SelectQuery` in a DataMap is now stored as a single query String in the syntax of `ObjectSelect.parse(..)`, instead
of the root attributes and the separate `qualifier`, `ordering` and `prefetch` elements. Projects are upgraded
automatically, either in memory by the runtime, or permanently by CayenneModeler.
* `SelectQueryDescriptor` now holds the query String as is (`getQl()` / `setQl(..)`), and derives the rest from it.
The query String can define any `ObjectSelect` or `ColumnSelect`, including column queries, `having` and queries
rooted in a DbEntity (`from db:ARTIST`). Its root is defined by the "from" clause, so `setRoot(..)` is not supported.
The setters of the query parts (`setQualifier(..)`, `addOrdering(..)`, `addPrefetch(..)`, `setDistinct(..)`, etc.)
were removed and their getters are deprecated, use `getParsedQuery()` instead. `buildQuery(..)` returns a
`FluentSelect`, not an `ObjectSelect`. A query String that can't be parsed no longer fails the DataMap loading, but
the execution of the query, and is reported by the project validator. CayenneModeler edits a mapped `ObjectSelect`
as a query String.
* Per [CAY-3038](https://issues.apache.org/jira/browse/CAY-3038) `EJBQLQuery` was removed. The replacement is `ObjectSelect` / `ColumnSelect` parsed from a String
(see "Queries Parsed from Strings" above). As there is no automatic conversion of EJBQL to the new query language,
an `EJBQLQuery` stored in a DataMap is dropped by the upgrade to project version 14 with a warning, and must be
Expand Down
270 changes: 163 additions & 107 deletions cayenne-cgen/src/main/java/org/apache/cayenne/gen/DataMapUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,29 +19,36 @@

package org.apache.cayenne.gen;

import org.apache.cayenne.DataRow;
import org.apache.cayenne.exp.Expression;
import org.apache.cayenne.exp.ExpressionException;
import org.apache.cayenne.exp.ExpressionParameter;
import org.apache.cayenne.exp.InExp;
import org.apache.cayenne.exp.ListExp;
import org.apache.cayenne.exp.NotInExp;
import org.apache.cayenne.exp.ObjPathExp;
import org.apache.cayenne.map.DataMap;
import org.apache.cayenne.map.DbEntity;
import org.apache.cayenne.map.ObjAttribute;
import org.apache.cayenne.map.ObjEntity;
import org.apache.cayenne.map.ObjRelationship;
import org.apache.cayenne.map.QueryDescriptor;
import org.apache.cayenne.map.SelectQueryDescriptor;
import org.apache.cayenne.query.Ordering;
import org.apache.cayenne.projecttools.validation.SelectQueryPaths;
import org.apache.cayenne.projecttools.validation.SelectQueryPaths.Clause;
import org.apache.cayenne.projecttools.validation.SelectQueryPaths.PathRef;
import org.apache.cayenne.query.FluentSelect;
import org.apache.cayenne.util.Util;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.LinkedList;
import java.util.HashSet;
import java.util.IdentityHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import javax.lang.model.SourceVersion;

/**
* Attributes and Methods for working with Queries.
Expand All @@ -50,7 +57,14 @@
*/
public class DataMapUtils {

Map<String, Map<String, String>> queriesMap = new HashMap<>();
private static final String OBJECT_TYPE = Object.class.getName();

// the names of the variables of the generated query methods
private static final Set<String> RESERVED_NAMES = Set.of("context", "query");

// the parameters are requested many times per query by the templates, the identity of the parsed query detects a
// change of the query String
private final Map<SelectQueryDescriptor, CachedParameters> parameters = new IdentityHashMap<>();

/**
* Return valid method name based on query name (replace all illegal
Expand All @@ -64,144 +78,186 @@ public String getQueryMethodName(QueryDescriptor query) {
}

/**
* Get all parameter names that used in query qualifier.
* Get all parameter names used in the query, in the order of their first occurrence. The names are the Java names
* of the parameters, see {@link QueryParameter#javaName()}.
*
* @param query select query descriptor
* @return Parameter names.
* @deprecated since 5.0, use {@link #getParameters(SelectQueryDescriptor)}, that provides both the names to bind
* the parameters by and their Java names
*/
@Deprecated(since = "5.0")
public Collection<String> getParameterNames(SelectQueryDescriptor query) {

if (query.getQualifier() == null) {
return Collections.emptySet();
}

Map<String, String> queryParameters = queriesMap.get(query.getName());

if (queryParameters == null) {
queryParameters = getParameterNames(query.getQualifier(), query.getRoot());
queriesMap.put(query.getName(), queryParameters);
}

return parseQualifier(query.getQualifier().toString());
return getParameters(query).stream().map(QueryParameter::javaName).toList();
}

public Boolean isValidParameterNames(SelectQueryDescriptor query) {

if (query.getQualifier() == null) {
return true;
}

Map<String, String> queryParameters = queriesMap.get(query.getName());

if (queryParameters == null) {
try {
queryParameters = getParameterNames(query.getQualifier(), query.getRoot());
} catch (Exception e) {
// if we have wrong path in queryParameters return false.
return false;
}
}

for (Ordering ordering : query.getOrderings()) {
// validate paths in ordering
String path = ordering.getSortSpecString();
try {
((ObjEntity) query.getRoot()).resolvePath(path);
} catch (ExpressionException e) {
// if we have wrong path in orderings return false.
return false;
}
/**
* Returns the "$name" parameters of the query, in the order of their first occurrence, each with the name to bind
* it by, a unique Java identifier to use for it in the generated code, and its type.
*
* @since 5.0
*/
public List<QueryParameter> getParameters(SelectQueryDescriptor query) {
FluentSelect<?, ?> parsed = query.isValid() ? query.getParsedQuery() : null;
CachedParameters cached = parameters.get(query);
if (cached == null || cached.parsed() != parsed) {
cached = new CachedParameters(parsed, parsed != null ? computeParameters(query) : List.of());
parameters.put(query, cached);
}

return true;
return cached.parameters();
}

/**
* Get list of parameter names in the same order as in qualifier.
*
* @param qualifierString
* to be parsed
* @return List of parameter names.
* Returns whether a method can be generated for the query: the query String can be parsed, its root is mapped,
* and the entity names and paths of all its clauses can be resolved.
*/
private Set<String> parseQualifier(String qualifierString) {
Set<String> result = new LinkedHashSet<>();
Pattern pattern = Pattern.compile("\\$[\\w]+");
Matcher matcher = pattern.matcher(qualifierString);
while (matcher.find()) {
String name = matcher.group();
result.add(Util.underscoredToJava(name.substring(1), false));
public Boolean isValidParameterNames(SelectQueryDescriptor query) {
if (!query.isValid() || !(query.getRoot() instanceof ObjEntity || query.getRoot() instanceof DbEntity)) {
return false;
}

return result;
return SelectQueryPaths.check(query).isEmpty();
}

public boolean hasParameters(SelectQueryDescriptor query) {
Map<String, String> queryParameters = queriesMap.get(query.getName());

if (queryParameters == null) {
return false;
}

return queryParameters.keySet().size() > 0;

return !getParameters(query).isEmpty();
}

/**
* Get type of parameter for given name.
*
* @param query descriptor
* @param name parameter name
* @param name the Java name of the parameter, see {@link QueryParameter#javaName()}
* @return Parameter type.
* @deprecated since 5.0, use {@link #getParameters(SelectQueryDescriptor)}
*/
@Deprecated(since = "5.0")
public String getParameterType(SelectQueryDescriptor query, String name) {
return queriesMap.get(query.getName()).get(name);
for (QueryParameter parameter : getParameters(query)) {
if (parameter.javaName().equals(name)) {
return parameter.type();
}
}
return null;
}

/**
* Returns the fully qualified name of the type of the objects that the query fetches: the class of the root
* ObjEntity, a DataRow, an Object for a single column, or an Object[] for multiple columns.
*
* @since 5.0
*/
public String getResultType(SelectQueryDescriptor query) {
return switch (query.getResultKind()) {
case ENTITY -> ((ObjEntity) query.getRoot()).getClassName();
case SCALAR -> OBJECT_TYPE;
case ARRAY -> Object[].class.getCanonicalName();
case DATA_ROW -> DataRow.class.getName();
// a query with an unmapped root can't be run, but gets a type anyway
case null -> DataRow.class.getName();
};
}

private Map<String, String> getParameterNames(Expression expression, Object root) {
if (expression != null) {
Map<String, String> types = new HashMap<>();
String typeName = "";
List<String> names = new LinkedList<>();
private List<QueryParameter> computeParameters(SelectQueryDescriptor query) {
Map<String, String> types = parameterTypes(query.getDataMap(), query.getParsedQuery());

for (int i = 0; i < expression.getOperandCount(); i++) {
Object operand = expression.getOperand(i);
// the Java names must be unique and must not clash with the keywords and the variables of the generated method
Set<String> javaNames = new HashSet<>(RESERVED_NAMES);
List<QueryParameter> result = new ArrayList<>();
for (String name : query.getParameterNames()) {
String javaName = uniqueJavaName(name, javaNames);
result.add(new QueryParameter(name, javaName, types.getOrDefault(name, OBJECT_TYPE)));
}
return List.copyOf(result);
}

if (operand instanceof Expression) {
types.putAll(getParameterNames((Expression) operand, root));
}
private static String uniqueJavaName(String name, Set<String> taken) {
String base = Util.underscoredToJava(name, false);
if (base.isEmpty() || !Character.isJavaIdentifierStart(base.charAt(0))) {
base = "p" + base;
}

if (operand instanceof ObjPathExp pathExp) {
Object component = ((ObjEntity) root)
.resolvePath(pathExp.getExpandedPath())
.getLast();
typeName = switch (component) {
case ObjAttribute attribute -> attribute.getType();
case ObjRelationship relationship -> relationship.getTargetEntity().getClassName();
default -> "Object";
};
}
String javaName = base;
for (int i = 2; SourceVersion.isKeyword(javaName) || !taken.add(javaName); i++) {
javaName = base + i;
}
return javaName;
}

if (operand instanceof ListExp) {
Object[] values = (Object[]) ((ListExp) operand).getOperand(0);
for (Object value : values) {
if (value instanceof ExpressionParameter) {
names.add(((ExpressionParameter) value).getName());
private static Map<String, String> parameterTypes(DataMap dataMap, FluentSelect<?, ?> query) {
Map<String, String> types = new HashMap<>();
if (dataMap == null) {
return types;
}

// a parameter takes the type of a path compared to it, i.e. of the last path among the operands of the
// expression that has the parameter as its operand
Map<Expression, String> pathTypes = new IdentityHashMap<>();
SelectQueryPaths.walk(dataMap, query, path -> {
if ((path.clause() == Clause.WHERE || path.clause() == Clause.HAVING)
&& path.path() instanceof ObjPathExp && path.parent() != null) {
pathTypes.put(path.parent(), pathType(path));
}
});

pathTypes.forEach((expression, typeName) -> {
// the right-hand side parameter of "in" is a collection of the values of the path
boolean in = expression instanceof InExp || expression instanceof NotInExp;

for (int i = 0; i < expression.getOperandCount(); i++) {
switch (expression.getOperand(i)) {
case ExpressionParameter parameter -> {
String parameterType = in && i == 1 ? collectionType(typeName) : typeName;
types.merge(parameter.getName(), parameterType, DataMapUtils::moreSpecific);
}
case ListExp list -> {
for (Object value : (Object[]) list.getOperand(0)) {
if (value instanceof ExpressionParameter parameter) {
types.merge(parameter.getName(), typeName, DataMapUtils::moreSpecific);
}
}
}
case null, default -> {
}
}
}
});
return types;
}

if (operand instanceof ExpressionParameter) {
names.add(((ExpressionParameter) operand).getName());
}
// the type of the attribute or relationship at the end of a path, or Object if the path can't be resolved. An
// invalid path is a problem of the query, reported by the validator, and not of the type inference
private static String pathType(PathRef path) {
if (path.components() == null) {
return OBJECT_TYPE;
}

}
return switch (path.components().getLast()) {
case ObjAttribute attribute -> attribute.getType();
case ObjRelationship relationship -> relationship.getTargetEntity().getClassName();
default -> OBJECT_TYPE;
};
}

for (String name : names) {
types.put(Util.underscoredToJava(name, false), typeName);
}
private static String collectionType(String elementType) {
return Collection.class.getName() + "<" + elementType + ">";
}

return types;
private static String moreSpecific(String type1, String type2) {
if (OBJECT_TYPE.equals(type1) || type1.equals(type2)) {
return type2;
}
return Collections.emptyMap();
return OBJECT_TYPE.equals(type2) ? type1 : OBJECT_TYPE;
}

private record CachedParameters(FluentSelect<?, ?> parsed, List<QueryParameter> parameters) {
}

/**
* A "$name" parameter of a select query.
*
* @since 5.0
*/
public record QueryParameter(String name, String javaName, String type) {
}
}
12 changes: 11 additions & 1 deletion cayenne-cgen/src/main/java/org/apache/cayenne/gen/ImportUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

Expand Down Expand Up @@ -108,9 +109,18 @@ public void addReservedType(String typeName) {
* org.apache.cayenne.GenericPersistentObject.
*
* @param typeName
* FQ data type class name.
* FQ data type class name. For a parameterized type, like
* "java.util.List&lt;java.lang.String&gt;", all the classes in it are registered.
*/
public void addType(String typeName) {
if (typeName != null && typeName.indexOf('<') >= 0) {
Matcher matcher = StringUtils.CLASS_NAME.matcher(typeName);
while (matcher.find()) {
addType(matcher.group());
}
return;
}

if (!canRegisterType(typeName)) {
return;
}
Expand Down
Loading
Loading